Method and system for cleaning big language model training pictures

Abstract

The invention relates to the technical field of large language models, in particular to a method and system for cleaning large language model training pictures, comprising the steps of automatically checking and deleting damaged images, loading and processing images to delete blurred images, processing tasks in parallel and adopting streaming processing, deleting repeated images, and finally verifying and storing. The beneficial effects are that automatic cleaning of image data is realized, manual intervention is reduced, and processing efficiency is improved. Through operations of denoising, deblurring, format unification and the like, the image quality is remarkably improved, and the model training effect is enhanced. The computing power resource utilization rate is improved, and resource waste caused by low-quality data is avoided. The method is suitable for various large language model training scenes, and parameters and processes can be flexibly adjusted according to requirements.

Description

Technical Field

[0001] This invention relates to the field of large language model technology, specifically to a method and system for cleaning training images for large language models. Background Technology

[0002] Large language models are the cornerstone of artificial intelligence, and the quantity and quality of pre-training data are the main factors affecting the accuracy and efficiency of large language models.

[0003] In practical applications, the image data acquired by trainers often suffers from noise, blurring, corruption, and formatting issues, severely impacting the feedback performance of large models. Therefore, efficient image cleaning is an indispensable part of building efficient computer vision systems. OpenCV provides rich image processing capabilities; by performing a series of processes such as denoising, deblurring, and color correction on images, it can not only significantly improve image quality but also enhance the robustness and accuracy of the model.

[0004] Therefore, a method and system for cleaning large language model training images are needed, which can achieve automated screening and cleaning of massive pre-trained image data by adding configurations and calling OpenCV. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for cleaning large language model training images to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a method for cleaning large language model training images, comprising the steps of automatically checking and deleting damaged images:

[0007] By adding configuration, Pillow tools can be used to quickly detect image integrity, automatically removing incomplete or corrupted images that cannot be loaded, keeping only the images that pass the detection and their paths, and then resaving the images and paths that pass the detection.

[0008] Preferably, it also includes the step of loading and processing the image to remove the blurry image:

[0009] The processed images are processed by OpenCV. A blur threshold is set, the Laplacian variance of the image is calculated, the calculation result is compared with the threshold, and the sharpness of the image is evaluated by the gradient of the comparison result. Only high-quality images that meet the sharpness requirements are retained, and blurry images with low sharpness that may interfere with model training are deleted.

[0010] Preferably, it also includes the step of parallel processing of tasks and using streaming processing:

[0011] In the image processing process, parallel processing is adopted, and streaming processing technology is used to avoid loading all images at once, thereby improving processing efficiency and optimizing system resource utilization.

[0012] Preferably, the method also includes the step of deleting duplicate images:

[0013] Image detection is performed based on image hash values ​​to identify and remove duplicate images, preventing duplicate images from causing the model to overfit certain samples and ensuring the diversity and effectiveness of training data.

[0014] Preferably, the process also includes final verification and storage steps:

[0015] After completing the steps of automatically checking and deleting damaged images, deleting blurry images, parallel processing tasks, and deleting duplicate images, the cleaned image list is finally verified to ensure that the image quality meets the requirements. Then, the cleaned image list is saved to a file to provide data support for subsequent professional model training.

[0016] A system for cleaning training images of a large language model includes an image integrity detection and deletion module: This module, through configuration, uses the Pillow (PIL) tool to quickly detect the integrity of the input images. For images that cannot be loaded due to incompleteness or file corruption, it automatically detects and removes them. At the same time, it re-saves the images that pass the detection and their corresponding paths to ensure the integrity of the images used in subsequent processing.

[0017] Preferably, it also includes an image blur detection and deletion module: This module receives the image processed by the image integrity detection and deletion module, loads and processes the image using OpenCV; a blur threshold is preset, the Laplacian variance of the image is calculated, and the calculation result is compared with the preset threshold. The image sharpness is evaluated based on the gradient of the image comparison result, and only high-quality images that meet the sharpness requirements are retained, while blurry images with low sharpness that may interfere with model training are deleted.

[0018] Preferably, it also includes a parallel streaming processing module: this module adopts a parallel processing approach and uses streaming processing technology during image processing to avoid loading all images at once; through reasonable task allocation and data scheduling, multiple image processing tasks are executed in parallel, improving the overall processing efficiency of the system, optimizing the utilization of system resources, and ensuring the smoothness and efficiency of the image processing workflow.

[0019] Preferably, it also includes a duplicate image detection and deletion module: this module performs duplicate detection on the images processed by the previous module based on the image hash value, identifies duplicate images by calculating the hash value of each image and comparing the hash values; deletes the detected duplicate images to prevent duplicate images from causing the model to overfit certain samples, and ensures the diversity and effectiveness of the training data.

[0020] Preferably, it also includes a final verification and storage module: after the image integrity detection and deletion module, image blur detection and deletion module, parallel streaming processing module and duplicate image detection and deletion module have completed their respective processing, this module performs a final verification on the cleaned image list to ensure that the image quality meets the requirements for training large language models; after the verification is passed, the cleaned image list is saved to a specified file to provide reliable data support for subsequent professional model training.

[0021] Compared with the prior art, the beneficial effects of the present invention are:

[0022] This invention proposes a method and system for cleaning training images for large language models, achieving automated image data cleaning, reducing manual intervention, and improving processing efficiency. Through operations such as denoising, deblurring, and format unification, image quality is significantly improved, enhancing model training effects. It also improves the utilization of computing resources, avoiding resource waste caused by low-quality data. Applicable to various large language model training scenarios, parameters and processes can be flexibly adjusted according to requirements. Attached Figure Description

[0023] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the present invention clear and complete, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only some, not all, embodiments of the present invention, and are merely illustrative of the embodiments of the present invention. They are not intended to limit the embodiments of the present invention. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] Example 1, please refer to Figure 1 This invention provides a technical solution: a method for cleaning large language model training images, comprising the following steps:

[0026] 1. Automatically check and delete corrupted images

[0027] This invention, through the addition of configuration, automatically detects and removes images that may fail to load due to incompleteness or file corruption. The Pillow (PIL) tool can quickly detect image integrity, and images and paths that pass the detection are re-saved.

[0028] Completeness code check:

[0029]

[0030] 2. Load and process images, and remove blurry images.

[0031] The images processed in the previous step are then fed into OpenCV for further processing. (Blurred images refer to images with low sharpness, which may interfere with model training.) A blur threshold is set, the Laplacian variance of the image is calculated and compared with the threshold, and the sharpness is evaluated by the gradient of the comparison result. Only high-quality images are retained.

[0032] Blur detection:

[0033]

[0034] 3. Parallel processing tasks are performed using streaming processing to avoid loading all images at once.

[0035]

[0036] Parallel code:

[0037] 4. Delete duplicate images

[0038] Duplicate images can cause the model to overfit certain samples. We can detect duplicates based on image hash values.

[0039] 5. Final Verification and Storage

[0040] Finally, we save the cleaned image list to a file to provide data support for subsequent professional model training.

[0041] Example 2, based on Example 1, proposes a system for cleaning training images of large language models, including an image integrity detection and deletion module: This module, by adding configuration, uses the Pillow tool to quickly detect the integrity of the input images. For images that cannot be loaded due to incompleteness or file corruption, it automatically detects and removes them, while re-saving the images that pass the detection and their corresponding paths to ensure the integrity of the images used in subsequent processing.

[0042] It also includes an image blur detection and deletion module: This module receives the image after it has been processed by the image integrity detection and deletion module, loads and processes the image using OpenCV, presets a blur threshold, calculates the Laplacian variance of the image, compares the calculation result with the preset threshold, evaluates the image sharpness based on the gradient of the image comparison result, retains only high-quality images that meet the sharpness requirements, and deletes blurry images with low sharpness that may interfere with model training.

[0043] It also includes a parallel streaming processing module: This module uses parallel processing and streaming technology to avoid loading all images at once during image processing. Through reasonable task allocation and data scheduling, multiple image processing tasks can be executed in parallel, improving the overall processing efficiency of the system, optimizing system resource utilization, and ensuring the smoothness and efficiency of the image processing workflow.

[0044] It also includes a duplicate image detection and deletion module: This module detects duplicates in images processed by the previous module based on image hash values. It identifies duplicate images by calculating the hash value of each image and comparing the hash values; it deletes detected duplicate images to prevent the model from overfitting certain samples and to ensure the diversity and effectiveness of training data.

[0045] It also includes a final verification and storage module: After the image integrity detection and deletion module, image blur detection and deletion module, parallel streaming processing module, and duplicate image detection and deletion module have completed their respective processing, this module performs a final verification on the cleaned image list to ensure that the image quality meets the requirements for training large language models; after the verification is passed, the cleaned image list is saved to a specified file to provide reliable data support for subsequent professional model training.

[0046] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for cleaning training images for large language models, characterized in that: Includes steps for automatically checking and deleting corrupted images: By adding configuration, Pillow tools can be used to quickly detect image integrity, automatically removing incomplete or corrupted images that cannot be loaded, keeping only the images that pass the detection and their paths, and then resaving the images and paths that pass the detection.

2. The method for cleaning large language model training images according to claim 1, characterized in that: It also includes steps for loading and processing images to remove blurry images: The processed images are processed by OpenCV. A blur threshold is set, the Laplacian variance of the image is calculated, the calculation result is compared with the threshold, and the sharpness of the image is evaluated by the gradient of the comparison result. Only high-quality images that meet the sharpness requirements are retained, and blurry images with low sharpness that may interfere with model training are deleted.

3. The method for cleaning large language model training images according to claim 2, characterized in that: It also includes steps for parallel processing tasks and using streaming processing: In the image processing process, parallel processing is adopted, and streaming processing technology is used to avoid loading all images at once, thereby improving processing efficiency and optimizing system resource utilization.

4. The method for cleaning large language model training images according to claim 3, characterized in that: It also includes the step of removing duplicate images: Image detection is performed based on image hash values ​​to identify and remove duplicate images, preventing duplicate images from causing the model to overfit certain samples and ensuring the diversity and effectiveness of training data.

5. The method for cleaning large language model training images according to claim 4, characterized in that: It also includes the final verification and storage steps: After completing the steps of automatically checking and deleting damaged images, deleting blurry images, parallel processing tasks, and deleting duplicate images, the cleaned image list is finally verified to ensure that the image quality meets the requirements. Then, the cleaned image list is saved to a file to provide data support for subsequent professional model training.

6. A system for the method of cleaning large language model training images according to claim 5, characterized in that: Includes an image integrity detection and deletion module: This module, through configuration, uses Pillow tools to quickly detect the integrity of input images. For images that cannot be loaded due to incompleteness or file corruption, it automatically detects and removes them, while resaving the images that pass the detection and their corresponding paths to ensure the integrity of images used in subsequent processing.

7. The system according to claim 6, characterized in that: It also includes an image blur detection and removal module: this module receives the image after it has been processed by the image integrity detection and removal module, and uses OpenCV to load and process the image; A blur threshold is preset. The Laplacian variance of the image is calculated and compared with the preset threshold. The image sharpness is evaluated based on the gradient of the image comparison results. Only high-quality images that meet the sharpness requirements are retained, while blurry images with low sharpness that may interfere with model training are deleted.

8. The system according to claim 7, characterized in that: It also includes a parallel streaming processing module: This module uses parallel processing and streaming technology to avoid loading all images at once during image processing. Through reasonable task allocation and data scheduling, multiple image processing tasks can be executed in parallel, improving the overall processing efficiency of the system, optimizing system resource utilization, and ensuring the smoothness and efficiency of the image processing workflow.

9. A system according to claim 8, characterized in that: It also includes a duplicate image detection and deletion module: This module detects duplicates in images processed by the previous module based on image hash values. It identifies duplicate images by calculating the hash value of each image and comparing the hash values; it deletes detected duplicate images to prevent the model from overfitting certain samples and to ensure the diversity and effectiveness of training data.

10. A system according to claim 9, characterized in that: It also includes a final verification and storage module: After the image integrity detection and deletion module, image blur detection and deletion module, parallel streaming processing module, and duplicate image detection and deletion module have completed their respective processing, this module performs a final verification on the cleaned image list to ensure that the image quality meets the requirements for training large language models; after the verification is passed, the cleaned image list is saved to a specified file to provide reliable data support for subsequent professional model training.

Images