Deep learning model input image processing method and device, medium and equipment

By adjusting the features of the image to be input to make them close to the features of the training set, the problem of wasted computing power and insufficient adaptability during the training of deep learning model is solved, and resource saving and model performance improvement is achieved.

CN119006952AInactive Publication Date: 2024-11-22BEIJING JIAOTONG UNIV
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202411032470.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-30
Publication Date
2024-11-22
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The existing technology has the problem of wasting computing resources during deep learning model training. The general image enhancement method cannot adapt to different application scenarios, resulting in poor performance of the model in actual detection environment.

Method used

By obtaining the feature gap between the image to be input and the image in the training set, adjusting the parameters such as the brightness mean, contrast value and other parameters of the image to be input, making it closer to the features of the training set, thereby reducing the gap with the training set and inputting the adjusted image Execute tasks into deep learning models.

Benefits of technology

Effectively save computing resources, improve the adaptability and generalization capabilities of deep learning models, avoid the need for large-scale data set training, and reduce computing resources and time costs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119006952A_ABST
    Figure CN119006952A_ABST
Patent Text Reader

Abstract

The invention discloses a deep learning model input image processing method and device, a medium and equipment, and relates to the technical field of image processing. Comprising the following steps: acquiring a first parameter learned by a trained deep learning model and used for representing image features in a training set; acquiring a second parameter for representing the features of the to-be-input image; and obtaining a difference between the first parameter and the second parameter, and when the difference is greater than a set threshold value, performing adjustment of different feature dimensions on the to-be-input image to obtain a processed image. By adjusting the actually collected to-be-input image to be similar to the features in the training set, the processing method has the advantages that the amount of parameters needing to be adjusted is small, and the amount of computing resources needing to be paid is small; and the to-be-input image is subjected to targeted adjustment according to the training set, so that the adjustment amplitude of the to-be-input image can be ensured to be more adaptive to the trained deep learning model, and compared with a process of carrying out large-scale data set training on different scenes, computing power resources are further saved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method, device, medium and equipment for processing deep learning model input images. Background Art

[0002] Currently, the data set is usually enhanced before training the deep learning model to obtain a multi-state data set to help the deep learning model better generalize and adapt to different scenarios.

[0003] However, different application scenarios have different requirements for image quality and features. General image enhancement methods may not be suitable for all scenarios. Blindly applying image enhancement methods may not produce the expected results, resulting in a waste of computing resources. Summary of the invention

[0004] Based on this, in order to solve the problem of waste of computing power in the training process of deep learning models in the prior art, the present invention provides a deep learning model input image processing method, device, medium and equipment.

[0005] The present invention adopts the following technical solutions:

[0006] The present invention provides a deep learning model input image processing method, comprising:

[0007] A trained deep learning model is obtained, where the deep learning model is used to perform at least one of target detection, classification, recognition and segmentation tasks; a first parameter learned in the deep learning model training phase and used to characterize image features of images in a training set is obtained; wherein the data type of the first parameter is determined according to the task of the deep learning model in different application scenarios, and the first parameter is a vector composed of one or more of a brightness mean, a contrast value, a number of color channels and a picture size.

[0008] Obtain an image to be input to a deep learning model, and obtain a second parameter for characterizing image features of the image to be input to the deep learning model; wherein the second parameter has the same data type as the first parameter; obtain a gap between the first parameter and the second parameter, wherein the gap is obtained by calculating the Euclidean distance between the first parameter and the second parameter; when the gap is greater than a set threshold, adjust the first parameter of the image to be input to narrow the overall feature gap between the image to be input and the images in the training set, and obtain a processed image corresponding to the image to be input; wherein the processed image is input into a trained deep learning model to execute the task of the deep learning model.

[0009] When the first parameter and the second parameter are both brightness mean values, and the difference between the first parameter and the second parameter is greater than a set threshold, adjusting the brightness mean value of the input image includes:

[0010] Perform logarithmic operation on the brightness mean of the images in the training set and the brightness mean of the image to be input:

[0011]

[0012] Among them, L g (x, y) is the processed image, (x, y) is the pixel position in the processed image; L w (x,y) is the brightness of the pixel (x,y) of the image to be input. is the logarithmic mean of the brightness of the input image, is the logarithmic mean of the brightness of the images in the training set; log() represents the logarithmic operation.

[0013] When the first parameter and the second parameter are both contrast values, and the difference between the first parameter and the second parameter is greater than the set threshold, the adjustment of the contrast value of the input image includes:

[0014] Obtain the histogram of the images in the training set and the histogram of the image to be input;

[0015] The histogram of the input image is divided into three grayscale intervals according to the grayscale levels of the histogram of the image in the training set, and the number of pixels in the three grayscale intervals is approximately equal;

[0016] Histogram equalization is performed in three grayscale intervals respectively; the three grayscale intervals processed by histogram equalization are merged to obtain a processed image.

[0017] The present invention provides a deep learning model input image processing method and device, comprising:

[0018] An overall parameter acquisition module, used to acquire a first parameter learned in the deep learning model training phase for characterizing image features of images in a training set; wherein the data type of the first parameter is determined according to the task of the deep learning model in different application scenarios;

[0019] An input image parameter acquisition module, used to acquire a second parameter for characterizing the image feature of the image to be input to the deep learning model; wherein the second parameter has the same data type as the first parameter;

[0020] The input image processing module is used to obtain the gap between the first parameter and the second parameter. When the gap is greater than the set threshold, the first parameter of the input image is adjusted to narrow the overall feature gap between the input image and the image in the training set, and a processed image corresponding to the input image is obtained; wherein the processed image is input into the trained deep learning model to execute the task of the deep learning model.

[0021] The present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned deep learning model input image processing method.

[0022] The present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned deep learning model input image processing method when executing the program.

[0023] At least one of the above technical solutions adopted by the present invention can achieve the following beneficial effects:

[0024] In the method for processing the input image of the deep learning model provided by the present invention, the features of the input image are adjusted according to the gap between the features of the image to be input and the features of the images in the training set to narrow the gap between the processed image and the overall image of the training set.

[0025] Contrary to the traditional method of enhancing the data set to adapt to the actual scene, the method provided by the present invention is to adjust the features of the input image actually collected to be similar to the image features in the training set. This processing method makes targeted adjustments to the input image according to the training set. By changing the input image corresponding to different scenes to better adapt to the trained deep learning model, it can effectively meet the requirements of image quality in different application scenarios without the need for large-scale data set training process for different scenes, saving computing power resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0027] Figure 1 A schematic flowchart of the deep learning model input image processing method provided by the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in combination with the specific embodiments of the present invention and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] In practical applications, the target detection or recognition effect of deep learning models depends on the quality of the training data set. When the training data set is of high quality and highly similar to the target detection scene, the deep learning model can complete the target task well, and vice versa. For different scenes, the deep learning model needs to input a large amount of image data in the corresponding scene for large-scale training, and the input training set is mostly idealized, with unified image conditions, and single pictures, rather than actual target detection scenes where the lighting conditions change over time, the scenes are changeable, and the target attributes are not uniform. Therefore, the trained deep learning model has a certain inertia and can reason well in an environment similar to the training set, but if the actual detection environment is inconsistent with the training set environment, the target task execution result will be greatly affected, resulting in low accuracy or even unrecognizable situations.

[0030] To address this problem, most existing solutions focus on the training process, performing preliminary image enhancement operations on the training set, including adjusting the brightness, contrast, size, and angle of the image, so as to obtain a multi-state and multi-condition data set for training the model. The general technical solution is: (1) expanding on the basis of the original data set; (2) copying the data set instead of overwriting it; (3) low operation difficulty. In actual application, due to the exponential growth of the number of training sets, this simple image enhancement operation requires a lot of computing power resources, and different image enhancement methods will also achieve different results for different usage scenarios. If you want to experiment with a scenario, it will consume a lot of manpower costs, machine costs, and time costs. In addition, image enhancement has little effect on parameters such as model accuracy.

[0031] In this context, this embodiment provides a method for processing input images of a deep learning model, which adjusts the features of the input image according to the gap between the features of the image to be input and the features of the images in the training set, so as to narrow the gap between the processed image and the overall image in the training set. This method adjusts the actually collected input image to be similar to the features in the training set. This processing method not only requires a small number of parameters to be adjusted and requires a small amount of computing resources, but also makes targeted adjustments to the input image according to the training set to ensure that the adjustment range of the input image is more suitable for the trained deep learning model, which further saves computing resources compared to the large-scale data set training process for different scenarios.

[0032] Figure 1 The processing method flow of the deep learning model input image of this embodiment is shown. Figure 1 The method is described in detail and specifically comprises the following steps:

[0033] S1: Obtain a first parameter learned during the deep learning model training phase for characterizing image features in a training set; wherein the data type of the first parameter is determined according to the task of the deep learning model in different application scenarios.

[0034] The deep learning model can be used to perform tasks such as target detection, classification, recognition, prediction and segmentation. In the final task execution stage of the deep learning model, such as the prediction stage, the data such as the video and picture to be predicted are adaptively adjusted so that the trained model can be better applied to these data. This is the biggest difference between the method provided in this embodiment and the prior art. Specifically, within a specific time period, the first parameter learned in advance in the model training stage is read, and the first parameter is a vector composed of one or more of the brightness mean, contrast value, number of color channels and screen size.

[0035] S2: Obtain a second parameter for characterizing image features of an input image to a deep learning model; wherein the second parameter has the same data type as the first parameter.

[0036] The biggest difference between this method and the prior art is that the input image is adaptively adjusted in the final detection or prediction task execution stage so that the trained model can be better applied to these data. Therefore, it is necessary to obtain the second parameter of the input image, which has the same data type as the first parameter.

[0037] S3: Obtain the gap between the first parameter and the second parameter. When the gap is greater than a set threshold, adjust the first parameter of the input image to narrow the overall feature gap between the image to be input and the image in the training set, and obtain a processed image corresponding to the image to be input; wherein the processed image is input into the trained deep learning model to execute the task of the deep learning model.

[0038] In actual detection, the trained deep learning model may face the situation that the environment changes over time, such as lighting conditions, imaging environment, etc. These changes will affect the results of deep learning model task execution. Therefore, it is necessary to enhance the adaptability of target detection models to different environments. In practical applications, image enhancement methods have problems such as doubling of data sets, longer training time, and incompatibility between the use environment and the model. Specifically, after operations such as cropping, rotating, changing brightness, and adjusting contrast are completed, the original data set is still needed as the basis for training. Therefore, the data set after data enhancement will increase exponentially, and the more enhancement methods are selected, the larger the number of data sets will be. Similarly, the training time of deep learning models will increase with the increase in the number of data sets, and the data pool in the training process will increase, which will require higher computing power facilities such as graphics cards, making the entire training process lengthy and not conducive to rapid updates and iterations. However, it is impossible to fully adapt the actual detection environment to the data set used in the model training process, which will cause the model that obtains better results during testing to perform poorly in actual applications.

[0039] In response to the above problems, this embodiment makes the input image more in line with general conditions and closer to the training set for model training by adaptively adjusting the brightness, contrast, number of color channels and screen size of the input image. On the one hand, it can enhance the generalization ability of the deep learning model, and on the other hand, it eliminates the data enhancement method, which can avoid the shortcomings of large training sets, long training time and weak adaptability in the model training process, and improve the target detection capability more flexibly and effectively. The input image mentioned here can be the data to be annotated input in the prediction stage, and the adaptive adjustment refers to performing specific processing on the input image, matching the brightness and contrast of the picture in the training set through the module method in the deep learning model during the adjustment process, and adjusting the brightness, contrast and other parameters of the input image itself, so that its brightness will not be overexposed, too dark, etc. Specifically:

[0040] First, since the first parameter and the second parameter are actually in the form of parameter vectors, their gap or dissimilarity can be calculated by at least one of the Euclidean distance, Manhattan distance and cosine similarity. Then, the degree of difference between the image to be processed and the parameters of the model training is further determined. If it is greater than a specific threshold α, the adaptive adjustment processing of the image data is completed through open source tools such as OpenCV, otherwise it is directly input into the model for processing. Specifically, by adjusting the parameters such as brightness and contrast of the video or picture set to be processed, the image features such as the image brightness and contrast of the image are similar to a series of parameter features learned in advance in the model training stage, and the most reasonable adjustment values ​​of brightness and contrast are calculated, and the values ​​are assigned to all video frames or pictures to be processed. After the adjustment process is completed, the adjusted data to be processed will be output and the next processing flow will continue. The brightness, contrast, etc. of the adjusted data to be processed are closer to the data set of the training model. At the same time, through adaptive adjustment, overexposure and darkening in the data to be processed can be avoided, making the model more stable in prediction. More specifically:

[0041] The adjustment of the brightness feature dimension of the input image includes: when the first parameter and the second parameter are both brightness mean values, and the difference between the first parameter and the second parameter is greater than a set threshold, the adjustment of the brightness mean value of the input image includes:

[0042] Perform logarithmic operation on the brightness mean of the images in the training set and the brightness mean of the image to be input:

[0043]

[0044] Among them, L g (x, y) is the processed image, (x, y) is the pixel position in the processed image; L w (x,y) is the brightness of the pixel (x,y) of the image to be input. is the logarithmic mean of the brightness of the input image, is the logarithmic mean of the brightness of the images in the training set; log() represents the logarithmic operation.

[0045] The adjustment of the contrast feature dimension of the input image also includes, when the first parameter and the second parameter are both contrast values, and the difference between the first parameter and the second parameter is greater than a set threshold, the adjustment of the contrast value of the input image includes: obtaining the histogram of the image in the training set and the histogram of the image to be input; dividing the histogram of the image to be input into three grayscale intervals according to the grayscale of the histogram of the image in the training set, and the number of pixels in the three grayscale intervals is approximately equal; performing histogram equalization in the three grayscale intervals respectively; merging the three grayscale intervals processed by the histogram equalization to obtain a processed image.

[0046] Taking the trained YOLO (You Only Look Once) model as an example, after starting YOLO's prediction, that is, the annotation process, the data set to be detected, including a video or picture set, will be input. First, through the process of inputting the data set, the basic parameters of the data set to be detected can be obtained through various open source libraries in Python or C++, including the brightness mean and contrast value of each frame of the video, or the brightness mean and contrast value of each picture in the picture set; after obtaining these data, the corresponding parameters of the training set used in the adopted model will be read through the method provided by YOLO. By comparing the parameters of the data set to be predicted with the parameters of the training set, the optimal range of brightness or contrast of the data set to be predicted can be quickly obtained through statistical methods. After obtaining the range, the adaptive adjustment process of the data set to be predicted is started. Through the previously obtained brightness mean and contrast values, a value suitable for the data set to be detected can be obtained in the optimal range. This value can adapt to the environment of the training set and will not have too much impact on the data set to be detected, causing it to be overexposed or too dark. After completing the above processing process, the data set to be detected is input into the prediction method of YOLO for the next annotation operation. Through this adjustment process, not only the rationality of the set to be detected can be improved, but also the versatility of the model trained by YOLO can be improved. It is logically reasonable to adjust the set to be detected to a level that is suitable for the model itself. The processed image is input into the trained deep learning model to perform tasks such as target detection, classification, recognition, prediction, and segmentation.

[0047] like Figure 1 The processing method of the deep learning model input image shown in the figure innovatively proposes to adaptively adjust the data set to be detected to adapt to the task flow of the deep learning model. The change of the deep learning model workflow moves the traditional data enhancement operation of the training set to the back, which not only avoids the situation of redundant data sets, but also enhances the adaptability of the deep learning model. At the same time, for the input image, adaptive adjustment can purify the data set to a certain extent. With a small amount of computing power resources and time cost, the input image can be adjusted to a state that is convenient for the target task, and the experimental adjustment process that previously required a lot of time and computing power can be completed conveniently and quickly. This method requires a small number of parameters to be adjusted and requires less time cost. At the same time, since the introduction and calculation of external parameters are not involved in the adjustment process of the image, it can ensure that the adjustment range of the image is within a reasonable range and is more suitable for the target task of the deep learning model. The key technical points of the whole method include:

[0048] 1. Change the process of deep learning model execution tasks and change the detection process to: detect after adjusting the data to be processed.

[0049] 2. Read the training set data of the deep learning model, and use the brightness and contrast parameters of the training set as the reference range for adjusting the corresponding parameters of the input image; and perform adaptive brightness and contrast adjustments on the input image.

[0050] 3. Input the processed input image into the deep learning model to complete tasks such as target detection or classification.

[0051] The above is a method for processing a deep learning model input image provided by one or more embodiments of the present invention. Based on the same idea, the present invention also provides a corresponding deep learning model input image processing device, including:

[0052] The overall parameter acquisition module is used to obtain the first parameter learned in the deep learning model training phase for characterizing the image features of the images in the training set; wherein the data type of the first parameter is determined according to the task of the deep learning model in different application scenarios.

[0053] An input image parameter acquisition module is used to obtain a second parameter for characterizing the image features of the image to be input to the deep learning model; wherein the second parameter has the same data type as the first parameter.

[0054] The input image processing module is used to obtain the gap between the first parameter and the second parameter. When the gap is greater than the set threshold, the first parameter of the input image is adjusted to narrow the overall feature gap between the input image and the image in the training set, and a processed image corresponding to the input image is obtained; wherein the processed image is input into the trained deep learning model to execute the task of the deep learning model.

[0055] The task execution module is used to input the processed image into the trained deep learning model for processing.

[0056] For the specific definition of the processing device of the deep learning model input image, please refer to the definition of the processing method of the deep learning model input image above, which will not be repeated here. Each module in the above-mentioned processing device of the deep learning model input image can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0057] The present invention also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 Provides methods for processing images as input to deep learning models.

[0058] The present invention also provides a computer device structure. At the hardware level, the computer device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory. Of course, it may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 Provides methods for processing images as input to deep learning models.

[0059] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0060] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of the present invention.

Claims

1. A method for processing images input to a deep learning model, characterized in that: include: Acquire a first parameter learned in the deep learning model training phase for characterizing image features of images in a training set; wherein the data type of the first parameter is determined according to the task of the deep learning model in different application scenarios; Obtaining a second parameter for characterizing an image feature of an image to be input into a deep learning model; wherein the second parameter has the same data type as the first parameter; The difference between the first parameter and the second parameter is obtained. When the difference is greater than a set threshold, the first parameter of the input image is adjusted to narrow the overall feature difference between the input image and the images in the training set, and a processed image corresponding to the input image is obtained; wherein the processed image is input into the trained deep learning model to execute the task of the deep learning model.

2. The deep learning model input image processing as claimed in claim 1, characterized in that: The first parameter is a vector composed of one or more of a brightness mean value, a contrast value, a number of color channels and a picture size.

3. The deep learning model input image processing as claimed in claim 2, characterized in that: The adjusting of the first parameter of the input image specifically includes: When the first parameter and the second parameter are both brightness mean values, and the difference between the first parameter and the second parameter is greater than the set threshold, the adjustment of the brightness mean value of the input image includes: Perform logarithmic operation on the brightness mean of the images in the training set and the brightness mean of the image to be input: Among them, L g (x, y) is the processed image, (x, y) is the pixel position in the processed image; L w (x,y) is the brightness of the pixel (x,y) of the image to be input. is the logarithmic mean of the brightness of the input image, is the logarithmic mean of the brightness of the images in the training set; log() represents the logarithmic operation.

4. The deep learning model input image processing as claimed in claim 2, characterized in that: The adjusting of the first parameter of the input image specifically includes: When the first parameter and the second parameter are both contrast values, and the difference between the first parameter and the second parameter is greater than the set threshold, the adjustment of the contrast value of the input image includes: Obtain the histogram of the images in the training set and the histogram of the image to be input; The histogram of the input image is divided into three grayscale intervals according to the grayscale levels of the histogram of the image in the training set, and the number of pixels in the three grayscale intervals is approximately equal; Histogram equalization is performed in three grayscale intervals respectively; the three grayscale intervals processed by histogram equalization are merged to obtain a processed image.

5. The deep learning model input image processing as claimed in claim 1, characterized in that: The gap between the first parameter and the second parameter is obtained by calculating the Euclidean distance between the first parameter and the second parameter.

6. The deep learning model input image processing according to claim 1, characterized in that: The deep learning model is used to perform at least one of target detection, classification, recognition and segmentation tasks.

7. The deep learning model input image processing as claimed in claim 1, characterized in that: The deep learning model is a YOLO network model.

8. A deep learning model input image processing device, characterized in that: include: An overall parameter acquisition module, used to acquire a first parameter learned in the deep learning model training phase for characterizing image features of images in a training set; wherein the data type of the first parameter is determined according to the task of the deep learning model in different application scenarios; An input image parameter acquisition module, used to acquire a second parameter for characterizing the image feature of the image to be input to the deep learning model; wherein the second parameter has the same data type as the first parameter; The input image processing module is used to obtain the gap between the first parameter and the second parameter. When the gap is greater than the set threshold, the first parameter of the input image is adjusted to narrow the overall feature gap between the input image and the image in the training set, and a processed image corresponding to the input image is obtained; wherein the processed image is input into the trained deep learning model to execute the task of the deep learning model.

9. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer device, characterized in that The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Image feature-based image subject identification method

    CN107122375A

  • Image classification method based on subspace projection and dictionary learning

    CN109117860A

  • Data amplification method based on small sample scene

    CN113763234A

  • Method and apparatus for training instance segmentation model, and instance segmentation method

    WO2021238826A1