Automatic data cleaning method based on deep learning

Abstract

The invention discloses an automatic data cleaning method based on deep learning. The method comprises the steps of obtaining a to-be-cleaned image data set; marking the to-be-cleaned image data set to obtain a normal image data set and an abnormal image data set, and performing data enhancement on the data sets; constructing a generative adversarial network model, wherein the generative adversarial network model comprises a generator and a discriminator; performing adversarial training by using the marked normal image data set and the abnormal image data set, and training to generate a final adversarial network model; and inputting a to-be-detected image into the final adversarial network model, performing anomaly detection on the to-be-detected image to obtain an image anomaly detection result, and adopting different preset cleaning methods based on the image anomaly detection result. According to the method, the generative adversarial network model is utilized to realize anomaly detection and cleaning of the image data, so that the data processing efficiency can be greatly improved, and high-quality data support is provided for a large visual application model in the complex process industry.

Description

Technical Field

[0001] The present invention relates to the technical field of complex process industrial automation data cleaning, and in particular to an automated data cleaning method based on deep learning. Background Art

[0002] In complex process industries, the large amount of image data generated during the production process often has uneven data quality, noise, blur or abnormal data due to factors such as ambient light, equipment status changes, and operator interference. These low-quality data not only affect subsequent intelligent analysis and large model training, but also increase the complexity of data management. Traditional data cleaning methods usually rely on manually set rules and thresholds, which are difficult to adapt to the diverse data forms and changes in complex process industries, have low processing efficiency, and are difficult to ensure data accuracy and consistency. Summary of the invention

[0003] In view of the above problems, the present invention is proposed to provide an automated data cleaning method based on deep learning that overcomes the above problems or at least partially solves the above problems.

[0004] In order to solve the above technical problems, the embodiments of the present application disclose the following technical solutions:

[0005] The embodiment of the present invention discloses an automatic data cleaning method based on deep learning, comprising:

[0006] S100. Collect image data from a complex industrial production process using an industrial camera to obtain an image data set to be cleaned;

[0007] S200. Annotating the image data set to be cleaned to obtain a normal image data set and an abnormal image data set, and performing data enhancement on the normal image data set and the abnormal image data set;

[0008] S300. Construct a generative adversarial network model, wherein the generative adversarial network model includes a generator and a discriminator;

[0009] S400. Use the labeled normal image dataset and abnormal image dataset to perform adversarial training, adjust the hyperparameters to ensure the adversarial balance between the generator and the discriminator, and train and generate the final adversarial network model;

[0010] S500. Input the image to be detected into the final adversarial network model, perform anomaly detection on the image to be detected, obtain an image anomaly detection result, and adopt different preset cleaning methods based on the image anomaly detection result.

[0011] Furthermore, in S100, image data of a complex process industrial production process is collected by an industrial camera. The specific method includes: collecting images of different processes, ambient light and equipment operating conditions in a complex process industrial production process by an industrial camera, transmitting the images to an edge computing platform via an Internet of Things device, and preprocessing the images.

[0012] Furthermore, in S200, the image data set to be cleaned is annotated to obtain a normal image data set and an abnormal image data set, and data enhancement is performed on the normal image data set and the abnormal image data set; the specific method includes: annotating the image data set to be cleaned, marking abnormal images and normal images, and the abnormal images at least include blurred, damaged, noisy or texture-abnormal images; data enhancement is performed on the normal image data set and the abnormal image data set through operations such as random cropping, rotation, and brightness adjustment.

[0013] Furthermore, in S300, the generator is used to learn the distribution of normal images and generate images similar to real data for abnormal data repair and replacement; the generator introduces a lightweight convolution module, which adopts the depth-separable convolution technology to split the standard convolution operation into depth convolution and point convolution, thereby reducing the computational complexity.

[0014] Furthermore, in S300, the discriminator is used to detect whether the input image conforms to the normal image distribution. Through the adversarial learning between the discriminator and the generator, the discriminator has the ability to distinguish normal images from abnormal images. An attention mechanism is introduced into the discriminator. The attention mechanism improves the discriminator's ability to capture image details by automatically focusing on key areas in the image. The model's attention to different image areas can be dynamically adjusted, so that the discriminator can more accurately identify images with slight blur, noise or abnormal details.

[0015] Furthermore, in S400, adversarial training is performed using the labeled normal image dataset and abnormal image dataset, and the adversarial balance between the generator and the discriminator is ensured by adjusting the hyperparameters, and the final adversarial network model is generated by training. The specific method includes:

[0016] Training the discriminator: Input the normal image and the forged image generated by the generator into the discriminator. The discriminator extracts image features through the CNN network and learns to distinguish between real images and forged images. Through multiple iterations, the discriminator gradually enhances its ability to identify forged images;

[0017] Training the generator: The generator receives random noise as input and generates images that are similar to the real data distribution. As adversarial training proceeds, the generator gradually learns the distribution characteristics of normal data and generates more realistic images.

[0018] Adversarial training: The generator and the discriminator are trained through repeated adversarial iterations. The generation ability of the generator and the recognition ability of the discriminator are mutually enhanced, and finally an adversarial equilibrium state is reached.

[0019] Further, in S500, the image to be detected is input into the final adversarial network model, and anomaly detection is performed on the image to be detected to obtain an image anomaly detection result; the specific method includes:

[0020] When the image data to be detected is input into the model, the discriminator evaluates the image to determine whether the image conforms to the normal data distribution learned during training; the discriminator extracts the features of the image through its convolutional neural network structure, and uses the introduced attention mechanism to further enhance the ability to capture detail features; this mechanism enables the discriminator to pay more attention to key areas in the image and automatically adjust the attention to each part of the image, thereby improving the detection accuracy of image anomalies; each input image will obtain an anomaly score through the discriminator, and the higher the score, the greater the deviation of the image from the normal distribution; based on the score, it is automatically determined whether the image needs to be cleaned, and a basis is provided for subsequent cleaning strategies.

[0021] Furthermore, in S500, different preset cleaning methods are adopted based on the image anomaly detection result; the specific method includes: the image anomaly detection result includes severe abnormal data and mild abnormal data; when the image anomaly detection result is severe abnormal data, the model will automatically delete the severe abnormal image data, the deleted image will be marked as irreversible abnormal data, and the deletion operation will be recorded in the data management system to ensure the traceability of data processing.

[0022] Furthermore, when the image anomaly detection result is mild anomaly data, the generator will be used to automatically repair the mild anomaly data. The generator uses its learning ability of normal data distribution to generate a new image similar to the normal image to replace or repair the abnormal data. The generator generates a high-quality repaired image based on the input abnormal data to ensure that the repaired image meets normal visual features and maintains consistency with other images in the data set. The repaired image will replace the original mild anomaly data to ensure that the overall analysis effect of the data set will not be affected by minor anomalies.

[0023] Furthermore, an automated data cleaning method based on deep learning also includes: S600. Automatically managing the cleaned image data, recording the processing history of each image, including at least the cleaning method, anomaly detection results, and repair status; the image data is stored in a unified distributed storage system to support subsequent large model training and data analysis.

[0024] The beneficial effects of the above technical solution provided by the embodiment of the present invention include at least:

[0025] The present invention discloses an automatic data cleaning method based on deep learning, comprising: collecting image data in a complex process industrial production process through an industrial camera to obtain an image data set to be cleaned; annotating the image data set to be cleaned to obtain a normal image data set and an abnormal image data set, and performing data enhancement on the normal image data set and the abnormal image data set; constructing a generative adversarial network model, the generative adversarial network model including a generator and a discriminator; using the annotated normal image data set and the abnormal image data set for adversarial training, adjusting the hyperparameters to ensure the adversarial balance between the generator and the discriminator, and training to generate a final adversarial network model; inputting the image to be detected into the final adversarial network model, performing anomaly detection on the image to be detected, obtaining an image anomaly detection result, and taking preset different cleaning methods based on the image anomaly detection result. The present invention realizes anomaly detection and cleaning of image data by utilizing a generative adversarial network (GAN) model, and the present invention can greatly improve data processing efficiency and provide high-quality data support for large models of visual applications in complex process industries.

[0026] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0028] Figure 1 This is a flow chart of an automated data cleaning method based on deep learning in Example 1 of the present invention;

[0029] Figure 2 This is a flowchart of abnormal data detection and cleaning in Example 1 of the present invention. DETAILED DESCRIPTION

[0030] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0031] In order to solve the problems existing in the prior art, an embodiment of the present invention provides an automated data cleaning method based on deep learning.

[0032] Example 1

[0033] The present invention discloses an automated data cleaning method based on deep learning. Figure 1 , including: S100. Collecting image data in a complex process industrial production process through an industrial camera to obtain an image data set to be cleaned; in S100 of this embodiment, collecting image data in a complex process industrial production process through an industrial camera, the specific method includes: collecting images of different processes, ambient light and equipment operating conditions in a complex process industrial production process through an industrial camera, transmitting the images to an edge computing platform through an Internet of Things device, and preprocessing the images, and the preprocessing method includes an image cropping operation.

[0034] Specifically, industrial cameras are used to collect image data from complex process industrial production processes. In order to adapt to different processes, ambient light, and equipment operating conditions in industrial environments, the image acquisition system needs to combine the Internet of Things (IoT) technology to transmit the collected images to edge computing devices in real time. During data collection, the Internet of Things and edge computing technologies are used to achieve efficient data transmission and management, ensure the real-time and consistency of data, and provide basic support for automated detection of complex process industries. After the image data is transmitted, it is first preprocessed, including image cropping, to ensure that the data format input to the model is consistent and to improve the quality of the image data.

[0035] S200. Annotate the image dataset to be cleaned to obtain a normal image dataset and an abnormal image dataset, and perform data enhancement on the normal image dataset and the abnormal image dataset; In S200 of this embodiment, annotate the image dataset to be cleaned to obtain a normal image dataset and an abnormal image dataset, and perform data enhancement on the normal image dataset and the abnormal image dataset; The specific method includes: annotating the image dataset to be cleaned, marking abnormal images and normal images, and the abnormal images at least include blurred, damaged, noisy or texture abnormal images; perform data enhancement on the normal image dataset and the abnormal image dataset through random cropping, rotation, brightness adjustment and other operations to increase data diversity to improve the generalization ability of the GAN model. .

[0036] S300. Construct a generative adversarial network model, which includes a generator and a discriminator; in S300 of this embodiment, the generator is used to learn the distribution of normal images and generate images similar to real data for abnormal data repair and replacement; the generator introduces a lightweight convolution module, which uses a depth-separable convolution technology to split the standard convolution operation into depth convolution and point convolution, reducing the computational complexity.

[0037] Specifically, the main task of the generator is to learn and simulate the distribution of normal image data, and generate images that are very similar to real data by inputting random noise. In this embodiment, a lightweight convolution module is introduced in the design of the generator. This module simplifies the network structure and reduces the number of model parameters, so that the generator can significantly improve the operation speed while maintaining the generation quality. The module uses the depthwise separable convolution technology to split the standard convolution operation into depth convolution and point convolution, thereby reducing the computational complexity. Without affecting the generation quality, the number of parameters of the generator is reduced, the memory usage and the computational cost are reduced, so that it can adapt to the needs of large-scale image data processing.

[0038] The discriminator is used to detect whether the input image conforms to the normal image distribution. Through the adversarial learning between the discriminator and the generator, the discriminator has the ability to distinguish normal images from abnormal images. The attention mechanism is introduced into the discriminator, which automatically focuses on the key areas in the image to improve the discriminator's ability to capture image details. It can dynamically adjust the model's attention to different image areas, so that the discriminator can more accurately identify images with slight blur, noise or abnormal details.

[0039] Specifically, the core task of the discriminator is to determine whether the image is abnormal data by distinguishing whether the input image conforms to the normal data distribution. In this embodiment, the discriminator is based on the convolutional neural network (CNN) structure, and analyzes the details and patterns of the input image by extracting the deep features of the image. In order to further improve the detection accuracy of the discriminator, an attention mechanism is introduced in the discriminator, which can enhance the sensitivity of the model to detail features. The attention mechanism improves the discriminator's ability to capture image details by automatically focusing on key areas in the image. It can dynamically adjust the model's attention to different image areas, so that the discriminator can more accurately identify images with slight blur, noise or abnormal details. In industrial scenarios, many abnormal data are not large-scale damage or distortion, but often manifest as local detail problems. The introduction of the attention mechanism effectively improves the detection ability of such abnormal data.

[0040] S400. Use the labeled normal image dataset and abnormal image dataset to perform adversarial training, adjust the hyperparameters to ensure the adversarial balance between the generator and the discriminator, and train and generate the final adversarial network model;

[0041] In S400 of this embodiment, adversarial training is performed using annotated normal image datasets and abnormal image datasets, and the adversarial balance between the generator and the discriminator is ensured by adjusting the hyperparameters to train and generate the final adversarial network model. The specific method includes:

[0042] Training the discriminator: Input the normal image and the forged image generated by the generator into the discriminator. The discriminator extracts image features through the CNN network and learns to distinguish between real images and forged images. Through multiple iterations, the discriminator gradually enhances its ability to identify forged images;

[0043] Training the generator: The generator receives random noise as input and generates images that are similar to the real data distribution. As adversarial training proceeds, the generator gradually learns the distribution characteristics of normal data and generates more realistic images.

[0044] Adversarial training: The generator and the discriminator are trained through repeated adversarial iterations. The generation ability of the generator and the recognition ability of the discriminator are mutually enhanced, and finally an adversarial equilibrium state is reached.

[0045] In order to ensure the best performance of the model, this embodiment has been tuned to determine the specific hyperparameter settings, including the maximum number of iterations of 5000, the learning rate of 0.0002, the batch size of 32, and the learning rate decay coefficient of 0.5, which decays by half after every 500 iterations. At the same time, the update ratio of the discriminator and the generator is 1:1 to ensure the balance between the two. These hyperparameter settings effectively improve the convergence speed and accuracy of the model during the training process, ensuring the best effect between the generator and the discriminator.

[0046] S500. Input the image to be detected into the final adversarial network model, perform anomaly detection on the image to be detected, obtain an image anomaly detection result, and adopt different preset cleaning methods based on the image anomaly detection result.

[0047] After the model training is completed, the image data is anomaly detected and cleaned. This process uses the trained discriminator and generator to automatically process the new image data to ensure the quality and consistency of the data. In S500 of this embodiment, the image to be detected is input into the final adversarial network model, and the image to be detected is anomaly detected to obtain the image anomaly detection result; the specific method includes:

[0048] When the image data to be detected is input into the model, the discriminator evaluates the image to determine whether the image conforms to the normal data distribution learned during training; the discriminator extracts the features of the image through its convolutional neural network structure, and uses the introduced attention mechanism to further enhance the ability to capture detail features; this mechanism enables the discriminator to pay more attention to key areas in the image and automatically adjust the attention to each part of the image, thereby improving the detection accuracy of image anomalies; each input image will obtain an anomaly score through the discriminator, and the higher the score, the greater the deviation of the image from the normal distribution; based on the score, it is automatically determined whether the image needs to be cleaned, and a basis is provided for subsequent cleaning strategies.

[0049] In S500 of this embodiment, different cleaning methods are preset based on the image anomaly detection results; the specific methods include: the image anomaly detection results include severe abnormal data and mild abnormal data; when the image anomaly detection result is severe abnormal data, the model will automatically delete the severe abnormal image data, mark the deleted image as unrecoverable abnormal data, and record the deletion operation in the data management system to ensure the traceability of data processing. Specifically, for image data identified as severe abnormal by the discriminator (such as severely damaged, completely blurred or distorted images), the system will adopt a deletion strategy. This type of data is usually severely distorted due to interference from external factors (such as sensor failure, data acquisition equipment damage, etc.), and its effectiveness cannot be restored by repair. The model will automatically delete these severely abnormal image data to avoid negative impacts on subsequent data analysis, model training or system performance. The deletion strategy is automatically executed by the system without human intervention, ensuring the efficiency and consistency of the cleaning process. The deleted image will be marked as "unrecoverable abnormal data" by the system, and the operation will be recorded in the data management system to ensure the traceability of data processing.

[0050] When the image anomaly detection result is mild abnormal data, the generator will be used to automatically repair the mild abnormal data. The generator generates a new image similar to the normal image to replace or repair the abnormal data through its learning ability of normal data distribution. The generator generates a high-quality repair image based on the input abnormal data to ensure that the repaired image meets the normal visual features and maintains consistency with other images in the data set. The repaired image will replace the original mild abnormal data to ensure that the overall analysis effect will not be affected by minor abnormalities in the data set. Specifically, for mild abnormal data (such as slight blur, partial noise interference, etc.) detected by the discriminator, the system will use the generator to automatically repair it. This type of abnormal data usually retains a lot of valid information, and its integrity can be restored by the generator, thereby avoiding deleting too much valid data and maintaining the integrity of the data set. The generator generates a new image similar to the normal image to replace or repair the abnormal data through its learning ability of normal data distribution. The repair process is automated. The generator generates a high-quality repair image based on the input abnormal data to ensure that the repaired image meets the normal visual features and maintains consistency with other images in the data set. The repaired images will replace the original slightly abnormal data, ensuring that the overall analysis effect will not be affected by minor anomalies in the data set. At the same time, this process minimizes data loss, retains a large amount of effective information, and improves the quality and integrity of the data.

[0051] In some preferred embodiments, an automated data cleaning method based on deep learning also includes: S600. Automatically managing the cleaned image data, recording the processing history of each image, including at least the cleaning method, anomaly detection results, and repair status; the image data is stored in a unified distributed storage system to support subsequent large model training and data analysis.

[0052] The present embodiment discloses an automated data cleaning method based on deep learning, including: collecting image data in a complex process industrial production process through an industrial camera to obtain an image data set to be cleaned; annotating the image data set to be cleaned to obtain a normal image data set and an abnormal image data set, and performing data enhancement on the normal image data set and the abnormal image data set; constructing a generative adversarial network model, the generative adversarial network model including a generator and a discriminator; using the annotated normal image data set and the abnormal image data set for adversarial training, adjusting the hyperparameters to ensure the adversarial balance between the generator and the discriminator, and training to generate a final adversarial network model; inputting the image to be detected into the final adversarial network model, performing anomaly detection on the image to be detected, obtaining an image anomaly detection result, and taking preset different cleaning methods based on the image anomaly detection result. The present invention realizes anomaly detection and cleaning of image data by utilizing a generative adversarial network (GAN) model, and the present invention can greatly improve data processing efficiency and provide high-quality data support for large models of visual applications in complex process industries.

[0053] It should be understood that the specific order or hierarchy of steps in the disclosed process is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process can be rearranged without departing from the scope of protection of the present disclosure. The attached method claims present the elements of the various steps in an exemplary order and are not intended to be limited to the specific order or hierarchy described.

[0054] In the above detailed description, various features are grouped together in a single embodiment to simplify the disclosure. This method of disclosure should not be interpreted as reflecting an intention that the embodiments of the claimed subject matter require more features than are clearly stated in each claim. On the contrary, as reflected in the appended claims, the invention is in a state of having less than all the features of the disclosed individual embodiments. Therefore, the appended claims are hereby expressly incorporated into the detailed description, with each claim standing on its own as a separate preferred embodiment of the invention.

[0055] Those skilled in the art will also appreciate that the various illustrative logic blocks, modules, circuits, and algorithmic steps described in conjunction with the embodiments herein can all be implemented as electronic hardware, computer software, or a combination thereof. In order to clearly illustrate the interchangeability between hardware and software, various illustrative components, blocks, modules, circuits, and steps are generally described above around their functions. Whether such functions are implemented as hardware or software depends on specific applications and the design constraints imposed on the entire system. A skilled person can implement the described functions in an alternative manner for each specific application, but such implementation decisions should not be interpreted as departing from the scope of protection of the present disclosure.

[0056] The steps of the method or algorithm described in conjunction with the embodiments herein may be directly embodied as hardware, a software module executed by a processor, or a combination thereof. The software module may be located in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a mobile disk, a CD-ROM, or any other form of storage medium well known in the art. An exemplary storage medium is connected to the processor so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium may also be an integral part of the processor. The processor and the storage medium may be located in an ASIC. The ASIC may be located in a user terminal. Of course, the processor and the storage medium may also be present in a user terminal as discrete components.

[0057] For software implementation, the techniques described in this application can be implemented with modules (e.g., procedures, functions, etc.) that perform the functions described in this application. These software codes can be stored in a memory unit and executed by a processor. The memory unit can be implemented within the processor or outside the processor. In the latter case, it is coupled to the processor in a communication manner via various means, which are well known in the art.

[0058] The above description includes examples of one or more embodiments. Of course, it is impossible to describe all possible combinations of components or methods for the purpose of describing the above embodiments, but it should be recognized by those skilled in the art that the various embodiments may be further combined and arranged. Therefore, the embodiments described herein are intended to cover all such changes, modifications and variations that fall within the scope of protection of the appended claims. In addition, with respect to the term "comprising" used in the specification or claims, the word is covered in a manner similar to the term "including", just as "including," is explained as a transitional word in the claims. In addition, any term "or" used in the specification of the claims is intended to mean "non-exclusive or".

Claims

1. An automated data cleaning method based on deep learning, characterized in that: include: S100. Collect image data from a complex industrial production process using an industrial camera to obtain an image data set to be cleaned; S200. Annotating the image data set to be cleaned to obtain a normal image data set and an abnormal image data set, and performing data enhancement on the normal image data set and the abnormal image data set; S300. Construct a generative adversarial network model, wherein the generative adversarial network model includes a generator and a discriminator; S400. Use the labeled normal image dataset and abnormal image dataset to perform adversarial training, adjust the hyperparameters to ensure the adversarial balance between the generator and the discriminator, and train and generate the final adversarial network model; S500. Input the image to be detected into the final adversarial network model, perform anomaly detection on the image to be detected, obtain an image anomaly detection result, and adopt different preset cleaning methods based on the image anomaly detection result.

2. The method for automatic data cleaning based on deep learning according to claim 1, characterized in that: In S100, image data of a complex process industrial production process is collected by an industrial camera. The specific method includes: collecting images of different processes, ambient light and equipment operating conditions in a complex process industrial production process by an industrial camera, transmitting the images to an edge computing platform via an IoT device, and preprocessing the images.

3. The method for automatic data cleaning based on deep learning according to claim 1, characterized in that: In S200, the image data set to be cleaned is annotated to obtain a normal image data set and an abnormal image data set, and data enhancement is performed on the normal image data set and the abnormal image data set; The specific method includes: annotating the image data set to be cleaned, marking abnormal images and normal images, wherein the abnormal images at least include blurred, damaged, noisy or texture-abnormal images; Data enhancement is performed on the normal image dataset and the abnormal image dataset through operations such as random cropping, rotation, and brightness adjustment.

4. The method for automatic data cleaning based on deep learning according to claim 1, characterized in that: In S300, the generator is used to learn the distribution of normal images and generate images similar to real data for abnormal data repair and replacement; the generator introduces a lightweight convolution module, which uses a depth-separable convolution technique to split the standard convolution operation into depth convolution and point convolution, thereby reducing computational complexity.

5. The method for automatic data cleaning based on deep learning according to claim 1, characterized in that: In S300, the discriminator is used to detect whether the input image conforms to the normal image distribution. Through the adversarial learning between the discriminator and the generator, the discriminator has the ability to distinguish normal images from abnormal images. An attention mechanism is introduced into the discriminator. The attention mechanism improves the discriminator's ability to capture image details by automatically focusing on key areas in the image. The model's attention to different image areas can be dynamically adjusted, so that the discriminator can more accurately identify images with slight blur, noise or abnormal details.

6. The method for automatic data cleaning based on deep learning according to claim 1, characterized in that: In S400, adversarial training is performed using the labeled normal image dataset and abnormal image dataset, and the adversarial balance between the generator and the discriminator is ensured by adjusting the hyperparameters, and the final adversarial network model is generated by training. The specific method includes: Training the discriminator: Input the normal image and the forged image generated by the generator into the discriminator together; the discriminator extracts image features through the CNN network and learns to distinguish between real images and forged images. Through multiple iterations, the discriminator gradually enhances its ability to identify forged images; Training the generator: The generator receives random noise as input and generates images that are similar to the real data distribution. As adversarial training proceeds, the generator gradually learns the distribution characteristics of normal data and generates more realistic images. Adversarial training: The generator and the discriminator are trained through repeated adversarial iterations. The generation ability of the generator and the recognition ability of the discriminator are mutually enhanced, and finally an adversarial equilibrium state is reached.

7. The method for automatic data cleaning based on deep learning according to claim 1, characterized in that: In S500, the image to be detected is input into the final adversarial network model, and anomaly detection is performed on the image to be detected to obtain an image anomaly detection result; the specific method includes: When the image data to be detected is input into the model, the discriminator evaluates the image to determine whether the image conforms to the normal data distribution learned during training; the discriminator extracts the features of the image through its convolutional neural network structure, and uses the introduced attention mechanism to further enhance the ability to capture detail features; this mechanism enables the discriminator to pay more attention to key areas in the image and automatically adjust the attention to each part of the image, thereby improving the detection accuracy of image anomalies; each input image will obtain an anomaly score through the discriminator, and the higher the score, the greater the deviation of the image from the normal distribution; based on the score, it is automatically determined whether the image needs to be cleaned, and a basis is provided for subsequent cleaning strategies.

8. The method for automatic data cleaning based on deep learning according to claim 1, characterized in that: In S500, different preset cleaning methods are adopted based on the image anomaly detection result; the specific method includes: the image anomaly detection result includes serious abnormal data and mild abnormal data; when the image anomaly detection result is serious abnormal data, the model will automatically delete the serious abnormal image data, the deleted image will be marked as irreversible abnormal data, and the deletion operation will be recorded in the data management system to ensure the traceability of data processing.

9. The method for automatic data cleaning based on deep learning according to claim 8, characterized in that: When the image anomaly detection result is mild anomaly data, the generator will be used to automatically repair the mild anomaly data. The generator uses its learning ability of normal data distribution to generate new images similar to normal images to replace or repair the abnormal data. The generator generates high-quality repaired images based on the input abnormal data to ensure that the repaired images meet normal visual features and maintain consistency with other images in the data set. The repaired images will replace the original mild anomaly data to ensure that the overall analysis effect will not be affected by minor anomalies in the data set.

10. The method for automatic data cleaning based on deep learning according to claim 1, characterized in that: Also includes: S600. Automatically manage the cleaned image data and record the processing history of each image, including at least the cleaning method, abnormality detection results, and repair status; The image data is stored in a unified distributed storage system to support subsequent large model training and data analysis.

Images