Image analysis method and image analysis system
Through the joint training of pre-trained mask extraction model and image analysis model, pixel-level image enhancement is achieved, which solves the problems of overexposure and noise amplification of light sources caused by undifferentiated enhancement of the whole image in the prior art, and improves the image analysis model Performance and adaptability.
Patent Information
- Application Number
- CN202411151296.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-21
- Publication Date
- 2024-11-08
AI Technical Summary
The prior art performs indiscriminate enhancement of the entire image during the image enhancement process, resulting in excessive exposure of light sources and amplification of image noise in night scenes, making it difficult to ensure image details and cannot actively adapt to the needs of image analysis models.
Using pre-trained mask extraction model and image analysis model, the pixel-level image enhancement is achieved through the same loss function training, and a sequence of image enhancement operations are performed on the original image using the mask, including denoising, contrast adjustment, brightness adjustment and sharpening. to generate enhanced images suitable for image analysis model.
It effectively avoids the problems of overexposure of light sources and noise amplification, improves the performance of the image analysis model and adaptability to various scenes, and ensures the retention of image details.
Smart Images

Figure CN118918037A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to image analysis and machine learning technology, and in particular to an image analysis method and an image analysis system. Background Art
[0002] Image analysis is to extract meaningful information from images, which involves tasks such as object detection, object recognition, and distance (or depth) estimation. At present, these tasks are usually implemented by corresponding machine learning models. However, in some special cases, such as images taken in low-visibility scenes, the performance of the model may be affected. Therefore, in practice, image enhancement is needed to improve image quality and improve model performance.
[0003] Traditional image enhancement algorithms, such as histogram equalization and gamma correction, enhance the brightness of the entire image. However, for night scenes, the captured images are generally not bright enough and the overall pixel value is low, so it is difficult to ensure the image details after enhancement. In addition, night scenes are often interfered by light sources such as car lights and street lights, which can easily lead to the amplification of image noise and overexposure of light sources, which is ultimately not conducive to the performance of image analysis models.
[0004] In addition, there are some algorithms, such as maximum entropy threshold segmentation, which distinguishes the bright and dark areas of the image and then uses different functions to enhance the brightness of the bright and dark areas respectively. However, this method has some defects. First, the enhanced image may not be smooth, especially in the case of adjacent thresholds. Second, the parameters of the function are usually set based on human experience, which may result in the enhanced image not necessarily meeting the needs of the image analysis model to perform specific tasks.
[0005] There are also some deep learning methods that extract features from images, determine the parameters of the filter based on the extracted features, and then use the filter to enhance the entire image indiscriminately. However, this approach cannot guarantee that the filter parameters actively adapt to the image analysis model, and like the aforementioned traditional image enhancement algorithm, it is easy to amplify image noise and cause overexposure of the light source, which is not conducive to the performance of the image analysis model.
[0006] In view of the above technical challenges, there is a need for an image analysis system in which the image enhancement model used can achieve pixel-wise enhancement and actively adapt the image analysis model. Summary of the invention
[0007] An embodiment of the present disclosure provides an image analysis system, which includes a storage unit and a processing unit. The processing unit loads a program from the storage unit to execute an image enhancement module and an image analysis module. The image enhancement module involves inputting an original image into a pre-trained mask extraction model, obtaining a plurality of masks corresponding to the original image output by the mask extraction model, and performing a sequence of image enhancement operations on the original image using the masks to obtain an enhanced image. The image analysis module involves inputting an enhanced image into a pre-trained image analysis model, and obtaining an inference result output by the image analysis model. The mask extraction model and the image analysis model are trained using the same loss function.
[0008] The embodiment of the present disclosure further provides an image analysis system, which includes an image enhancement circuit and an image analysis circuit coupled to each other. The image enhancement circuit is configured to input an original image into a pre-trained mask extraction model, obtain a plurality of masks output by the mask extraction model corresponding to the original image, and use the masks to perform a sequence of image enhancement operations on the original image to obtain an enhanced image. The image analysis circuit is configured to input the enhanced image into a pre-trained image analysis model, and obtain an inference result output by the image analysis model. The mask extraction model and the image analysis model are trained using the same loss function.
[0009] The embodiment of the present disclosure further provides an image analysis method, which includes inputting an original image into a pre-trained mask extraction model to obtain a plurality of masks (masks) corresponding to the original image output by the mask extraction model. The method also includes using the mask to perform a sequence of image enhancement operations on the original image to obtain an enhanced image. The method also includes inputting the enhanced image into a pre-trained image analysis model to obtain an inference result output by the image analysis model. The mask extraction model and the image analysis model are trained using the same loss function. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] The present disclosure will be better understood from the following illustrative embodiments with accompanying drawings. In addition, it should be understood that in the flowchart of the present disclosure, the execution order of each block can be changed, and / or some blocks can be changed, deleted or merged.
[0011] Figure 1A It is a flow block diagram of an image enhancement method according to an embodiment of the present disclosure.
[0012] Figure 1B is corresponding to Figure 1A Schematic diagram of the process of image enhancement method.
[0013] Figure 2A A block diagram is shown of more detailed steps of using a mask to perform an image enhancement operation on an original image to obtain an enhanced image in one embodiment of the present disclosure.
[0014] Figure 2B Corresponding to Figure 2A , a flowchart showing more detailed steps of using a mask to perform an image enhancement operation on an original image to obtain an enhanced image.
[0015] Figure 3 is a system block diagram of an image analysis system according to an embodiment of the present disclosure.
[0016] Figure 4 is a system block diagram of an image analysis system 40 according to an embodiment of the present disclosure.
[0017]
Explanation of symbols
[0018] 10: Methods
[0019] S11-S13: Step 101: Original Image
[0020] 102: Mask Extraction Model
[0021] M1-Mn: Mask
[0022] O1-On: Image enhancement operation
[0023] 105: Enhanced Image
[0024] 106: Image Analysis Model
[0025] 107: Inference
[0026] 108: Loss Function
[0027] S21-S23: Steps
[0028] M21-M24: Steps
[0029] O21: Denoising operation
[0030] 201: Denoised image
[0031] O22: Contrast adjustment operation
[0032] O23: Brightness adjustment operation
[0033] 202: Adjusted image
[0034] O24: Sharpening operation
[0035] 30: System
[0036] 301: Processing unit
[0037] 302: Storage unit
[0038] 310:Procedure
[0039] 311: Image enhancement module
[0040] 312: Image analysis module
[0041] 40: System
[0042] 401: Image enhancement circuit
[0043] 402: Image analysis circuit DETAILED DESCRIPTION
[0044] The following description lists various embodiments of the present invention, but is not intended to limit the content of the present invention. The actual scope of the invention is defined by the claims.
[0045] In the various embodiments listed below, the same reference numerals will be used to represent the same or similar elements or components.
[0046] Serial numbers in this specification and the claims, such as "first", "second", etc., are only for the convenience of explanation and have no sequential relationship with each other.
[0047] The following description of the embodiments of the apparatus or system is also applicable to the embodiments of the method, and vice versa.
[0048] The inventors have discovered that the prior art performs indiscriminate enhancement of the entire image during image enhancement processing. Therefore, in night scenes with light sources such as street lights and car lights, it is easy to cause excessive enhancement of the light source area while enhancing the brightness of the dark area of the night image, and overexposure of the light source occurs. In addition, when the target object to be identified in the application scenario is similar to the background features (such as color, brightness or texture), the indiscriminate image enhancement of the entire image has a poor enhancement effect on the local area of the image, resulting in object detection performance that is not as good as expected. After research, the inventors proposed an image enhancement method and system that can achieve pixel-wise enhancement and actively adapt to the image analysis model to solve the above-mentioned problems of the prior art. In addition, the mask extraction model and image analysis model trained by the method disclosed in the present invention can be deployed on an edge device or a terminal device (for example, downloaded from the cloud or transplanted via a storage device). In addition to providing economic benefits by not requiring the purchase of additional hardware equipment, these edge devices or computers can run independently during the inference / application stage of the model without the need for high computing power assistance from the cloud, thus achieving the purpose of keeping corporate information confidential.
[0049] Figure 1A is a flow block diagram of an image enhancement method 10 according to an embodiment of the present disclosure. Figure 1B is corresponding to Figure 1A Please refer to the flowchart of the image enhancement method 10. Figure 1A and Figure 1B , in order to gain a better understanding of the embodiments of the present disclosure. Figure 1A As shown, the image enhancement method 10 includes steps S11-S13.
[0050] In step S11, the original image 101 is input into the pre-trained mask extraction model 102, and a plurality of masks M1-Mn corresponding to the original image 101 output by the mask extraction model 102 are obtained. Next, in step S12, a sequence of image enhancement operations O1-On are performed on the original image 101 using the masks M1-Mn to obtain an enhanced image 105.
[0051] Masks M1-Mn are parameter arrays with the same resolution as the original image 101. Each element of each parameter array is a parameter required for performing a specific transformation on the corresponding pixel of the original image 101 in the subsequent steps. In other words, each element of the mask Mi (i is any integer from 1 to n) indicates how the corresponding pixel of the original image 101 performs the image enhancement operation Oi corresponding to the mask Mi. More specifically, the elements of the i-th row and j-th column of the mask M1 are used to perform the image enhancement operation O1 on the pixels of the i-th row and j-th column of the original image 101, and the elements of the i-th row and j-th column of the mask M2 are used to perform the image enhancement operation O2 on the pixels of the i-th row and j-th column of the original image 101, and so on. Thereby, pixel-level image enhancement can be achieved in step S12, replacing the whole-image indiscriminate enhancement of the prior art.
[0052] In addition, it should be understood that the original image 101 may have multiple channels. A typical example is a color image defined in an RGB color space, which includes three channels: R (red), G (green), and B (blue). In step S12, a corresponding image enhancement operation may be performed on a specific channel of the original image 101, or the same image enhancement operation may be performed on each channel of the original image 101, but the present disclosure is not limited to this. For example, if each R / G / B channel is to be processed, the mask M1-Mn generated by the mask extraction model 102 can be further divided into M1R / M1G / M1B, M2R / M2G / M2B, ..., MnR / MnG / MnB, wherein the R / G / B following the mask M1-Mn corresponds to the R / G / B channel, respectively. Therefore, if only the G channel is subjected to image enhancement processing, the G channel image of the original image will be subjected to image enhancement operations O1, O2, ...On, respectively, according to the element indications of the masks M1G, M2G, ..., MnG. In addition, if only the R / G channels are enhanced (that is, the B channel image is not processed), the R / G channel images of the original image will be enhanced by image enhancement operations O1, O2, ...On respectively according to the element indications of the masks M1R / M1G, M2R / M2G, ..., MnR / MnG.
[0053] In step S13 , the enhanced image 105 is input into the pre-trained image analysis model 106 , and the inference result 107 output by the image analysis model 106 is obtained.
[0054] The image analysis model 106 can be any model for extracting meaningful information from an image, such as an object detection model, an object recognition model, and a distance (or depth) estimation model, but the present disclosure is not limited to this. Taking the application scenario of autonomous driving as an example, the original image 101 can be image data sent back by a sensor or camera mounted on a vehicle, for the image analysis model 106 to perform tasks related to environmental perception. The object detection model is used to detect and locate various objects on the road, such as vehicles, pedestrians, obstacles, etc., to assist the vehicle in planning the optimal driving path, avoid obstacles, and improve driving safety. When the image analysis model 106 is an object detection model, the inference result 107 outputted by it is the position and range information of the object in the original image 101, usually represented by a bounding box. The object recognition model further enhances the level of environmental understanding. It not only detects the existence of objects, but also classifies objects to support decision making. For example, the object recognition model can be used to identify: the type of vehicle coming from behind, such as an ambulance, a fire truck, or a general vehicle; the color of a traffic sign, such as a red light, a yellow light, or a green light; various road traffic signs, such as no parking, no entry, road construction, etc.; and various road traffic markings, such as lane lines and zebra crossings on the road surface. When the image analysis model 106 is an object recognition model, the inference result 107 outputted by it is the category of the object, such as whether the vehicle coming from behind in the original image 101 is an ambulance, a fire truck, or a general vehicle, or whether the traffic sign in the original image 101 is a red light, a yellow light, or a green light. The distance estimation model is used to estimate the distance or depth information between the vehicle and the surrounding objects to support the decision-making of automatic following (e.g., maintaining an appropriate distance from the vehicle in front) and automatic parking (e.g., avoiding collisions or collisions with surrounding vehicles or walls). When the image analysis model 106 is a distance estimation model, the inference result 107 outputted by it is the estimated distance between the camera and the object in the original image 101.
[0055] The machine learning algorithm used to build the image analysis model 106 depends on the task type of the image analysis model 106 . For example, when the image analysis model 106 is an object detection model, algorithms such as Faster R-CNN, YOLO (You Only Look Once), SSD (Single Shot MultiBox Detector), FPN (Feature Pyramid Networks) can be used to implement the training of the image analysis model 106; when the image analysis model 106 is an object recognition model, algorithms such as convolutional neural networks (CNN) can be used to implement feature extractors, and decision trees, logistic regression, naive Bayes, random forests, support vector machines (SVM) or fully-connected neural networks can be used as classifiers to implement the training of the image analysis model 106; when the image analysis model 106 is a distance estimation model, algorithms such as convolutional neural networks, multilayer perceptrons (MLP), recurrent neural networks (RNNs) can be used. The image analysis model 106 is trained by using algorithms such as Convolutional Recurrent Neural Network (RNN) and Convolutional Recurrent Neural Network (CRNN) as regressors. In addition, various variations of gradient descent (GD), such as the common stochastic gradient descent (SGD) or adaptive moment estimation (Adam), can be used to calculate the gradient during the model training process to determine the direction of model parameter optimization. For example, the weights of the neural network can be updated by backpropagation to minimize the loss value.
[0056] The training data set used to train the training mask extraction model 102 and the image analysis model 106 may include multiple training data (such as the aforementioned training images with overexposure of light sources such as night street lights or car lights, products with local small defects, objects close to the background, etc.), each of which includes a training image and label data corresponding to the training image. The type of label data depends on the task type of the image analysis model 106. For example, when the image analysis model 106 is an object detection model, the label data is the location and range information of the object in the training image, usually represented by a bounding box; when the image analysis model 106 is an object recognition model, the label data is the category of the object, such as the rear vehicle in the image is an ambulance, a fire truck or a general vehicle, or the traffic sign in the image is a red light, a yellow light or a green light; when the image analysis model 106 is a distance estimation model, the label data is the actual distance between the camera and the object in the training image.
[0057] like Figure 1B As shown, the mask extraction model 102 and the image analysis model 106 are trained using the same loss function. More specifically, the training of the mask extraction model 102 and the image analysis model 106 is end-to-end learning, and the same loss function 108 is used as the evaluation index and constraint condition of the mask extraction model 102 and the image analysis model 106. Thereby, the mask extraction model 102 can actively adapt to the image analysis model 106 without human intervention to adjust the parameters. In addition, the overall performance of the image analysis model 106 and its adaptability to various scenarios can also be effectively improved.
[0058] The loss function 108 is used to evaluate the gap between the model output during training and the label data as the ground truth, and its form also depends on the task type of the image analysis model 106. For example, when the image analysis model 106 is an object recognition model, the loss function 108 may be a loss function commonly used in classification tasks such as cross-entropy (CE) loss, contrastive loss, hinge loss or KL divergence; when the image analysis model 106 is a distance estimation model, the loss function 108 may be a loss function commonly used in regression tasks such as mean squared error (MSE), mean absolute error (MAE), Huber Loss or Log-Cosh Loss.
[0059] In one embodiment, the mask extraction model 102 is implemented by a convolutional neural network with a U-net structure, which includes an encoder and a decoder. The encoder includes multiple convolutional layers for extracting feature representations of the image, also known as feature maps. In addition to the convolutional layers, an appropriate number of pooling layers can also be configured to reduce the spatial size of the feature map and retain the most important feature information. During the encoding process, the spatial information is gradually reduced and the feature information is gradually increased. The decoder includes multiple upsampling layers for mapping the feature map generated by the encoder back to the spatial size of the original image 101. Each upsampling layer is paired with a deconvolution layer for feature reconstruction and edge smoothing. In addition, the feature map output by each upsampling layer is concatenated with the feature map of the encoder of the previous layer to take into account the spatial information and feature information of each level for reconstruction. During the decoding process, the number of channels of the feature map gradually decreases and the resolution gradually increases, and finally a mask M1-Mn having the same resolution as the original image 101 is generated.
[0060] In one embodiment, the image enhancement operations O1-On contain a denoising operation, and the masks M1-Mn also contain a mask corresponding to the denoising operation, hereinafter referred to as the "first mask". The denoising operation is used to remove noise in the image. In one implementation, each element in the first mask is a binary value, that is, 0 or 1, representing whether the pixel is a noise point. In another implementation, each element in the first mask will first pass through a sigmoid activation function, and then use 0.5 as a threshold to map the value to 0 or 1. For pixel points indicated as noise points by the first mask, a bilinear interpolation method can be used to replace the value of the point with a weighted average of the pixel values of its four neighboring pixels, where the weight is inversely proportional to the distance. Other interpolation methods may also be used for denoising, such as nearest-neighbor interpolation or bicubic interpolation, but the present disclosure is not limited to this.
[0061] In one embodiment, the image enhancement operations O1-On include a contrast adjustment operation, and the masks M1-Mn also include a mask corresponding to the contrast adjustment operation, hereinafter referred to as the "second mask". The contrast adjustment operation is used to adjust the contrast of the image, that is, the difference in brightness between the images. In one implementation, the contrast adjustment operation involves multiplying the pixel value of the image by the value of the corresponding element in the second mask. When the value is greater than 1, it is equivalent to enhancing the contrast of the corresponding pixel; when the value is less than 1, it is equivalent to weakening the contrast of the corresponding pixel; when the value is equal to 1, it is equivalent to keeping the contrast of the corresponding pixel unchanged. In other implementations, different contrast adjustment functions, such as logarithmic transformation or exponential transformation, can be used to adjust the contrast, wherein the second mask serves as a parameter of the contrast adjustment function.
[0062] In one embodiment, the image enhancement operations O1-On include a brightness adjustment operation, and the masks M1-Mn also include a mask corresponding to the brightness adjustment operation, hereinafter referred to as the "third mask". The brightness adjustment operation is used to adjust the brightness of the image, that is, the brightness and darkness of the image. In one implementation, the brightness adjustment operation involves adding the pixel value of the image to the value of the corresponding element in the third mask. When the value is greater than 0, it is equivalent to increasing the brightness of the corresponding pixel point; when the value is less than 0, it is equivalent to reducing the brightness of the corresponding pixel point; when the value is equal to 0, it is equivalent to keeping the brightness of the corresponding pixel point unchanged. In other implementations, different brightness adjustment functions, such as logarithmic transformation or exponential transformation, can be used to adjust the brightness, where the third mask serves as a parameter of the brightness adjustment function.
[0063] In one implementation, both the contrast adjustment operation and the brightness adjustment operation use linear transformations. In other words, the contrast adjustment operation involves multiplying the pixel value of the image by the value of the corresponding element in the second mask, and the brightness adjustment operation involves adding the pixel value of the image to the value of the corresponding element in the third mask. Therefore, the contrast adjustment operation and the brightness adjustment operation can be performed together.
[0064] In one embodiment, the image enhancement operations O1-On include a sharpening operation, and the masks M1-Mn also include a mask corresponding to the sharpening operation, hereinafter referred to as the "fourth mask". The sharpening operation is used to enhance the visibility of edge contours in the image. In one implementation, the sharpening operation involves Gaussian blurring the image to obtain a blurred image, subtracting the blurred image from each pixel value of the original image to obtain a contrast image, multiplying the pixel value of the contrast image by the value of the corresponding element in the fourth mask, and then adding it to the pixel value of the original image. In other implementations, various filters can be used to enhance the high-frequency information in the image to achieve a sharpening effect, wherein the fourth mask serves as a parameter of the filter.
[0065] It should be understood that the aforementioned denoising operation, contrast adjustment operation, brightness adjustment operation and sharpening operation are only some implementations of the image enhancement operation, but the present disclosure is not limited thereto. In addition, the various embodiments of the present disclosure do not limit the execution order between the multiple image enhancement operations.
[0066] In one embodiment, the masks M1-Mn output by the mask extraction model 102 include the first mask, the second mask, the third mask and the fourth mask, and the image enhancement operations O1-On include the denoising operation, the contrast adjustment operation, the brightness adjustment operation and the sharpening operation. Figure 2A and Figure 2B This embodiment is described.
[0067] Figure 2A FIG. 4 is a flow chart showing more detailed steps of step S12 in an embodiment of the present disclosure. Figure 2B Corresponding to Figure 2A , which is a flow chart showing more detailed steps of step S12. Please refer to Figure 2A and Figure 2B , in order to gain a better understanding of the embodiments of the present disclosure. Figure 2A As shown, step S12 may further include steps S21-S23.
[0068] In step S21 , a denoising operation O21 is performed on the original image 101 using the first mask M21 , thereby obtaining a denoised image 201 .
[0069] In step S22 , the contrast adjustment operation O22 and the brightness adjustment operation O23 are respectively performed on the denoised image using the second mask M22 and the third mask M23 , thereby obtaining an adjusted image 202 .
[0070] In step S23 , a sharpening operation O24 is performed on the adjusted image 202 using a fourth mask M24 to obtain an enhanced image 105 .
[0071] In this embodiment, the brightness, contrast and sharpness of the original image 101 can be improved at the same time. This can not only ensure the recognition of details in the image, but also avoid the halo effect and noise amplification caused by excessive local enhancement, thereby ensuring the performance of the image analysis model 106.
[0072] In one embodiment, method 10 is only applied to images that need to be enhanced. In other words, only images that need to be enhanced are listed as original images 101. Whether an image needs to be enhanced can be determined by various known tools for evaluating image characteristics such as signal-to-noise ratio, contrast, brightness or sharpness, and the present disclosure is not limited to this. More specifically, in this embodiment, an image that meets any one (or more) of the following conditions can be listed as an original image 101: (1) the signal-to-noise ratio is lower than a specified signal-to-noise ratio threshold; (2) the contrast is lower than a specified contrast threshold; (3) the image contains an overexposed area, and the brightness value of each pixel in the overexposed area is higher than a specified brightness threshold; and / or (4) the sharpness is lower than a specified sharpness threshold.
[0073] Figure 3 FIG. 3 is a system block diagram of an image analysis system 30 according to an embodiment of the present disclosure. Figure 3 As shown, the system 30 may include a processing unit 301 and a storage unit 302 coupled to each other. The storage unit 302 stores a program 310, and the program 310 includes an image enhancement module 311 and an image analysis module 312. In addition, the storage unit may further store the aforementioned mask extraction model 102 and image analysis model 106.
[0074] The system 30 may be any computer system with computing capabilities, such as a personal computer (eg, a desktop computer or a notebook computer) or a server computer, or a mobile device such as a tablet computer or a smart phone, but the present disclosure is not limited thereto.
[0075] The processing unit 301 may include any one or more general or special processors and combinations thereof for executing instructions. In a typical embodiment, the processing unit may include a central processing unit (CPU) and a graphics processing unit (GPU), wherein the GPU is more efficient than the CPU in processing machine learning related tasks. Therefore, appropriate tasks can be assigned according to the characteristics of the CPU and the GPU, for example, tasks for obtaining image data or communicating with other devices are assigned to the CPU, while tasks related to image analysis and model training are assigned to the GPU. In a further embodiment, the processing unit 301 may also include a neural network processing unit (NPU) specifically optimized for deep learning. Compared to the GPU, the NPU may have an advantage in computing performance in operating deep neural networks, so tasks related to deep neural networks can be assigned to the NPU.
[0076] The storage unit 302 can be any device including a non-volatile memory (such as a read only memory, an electrically-erasable programmable read-only memory (EEPROM), a flash memory, a non-volatile random access memory (NVRAM)), such as a hard disk (HDD), a solid state drive (SSD) or an optical disk, but the present disclosure is not limited to this.
[0077] Program 310 is a sequence or a set of instructions for execution by a computer system. In various embodiments, program 310 may be written in any one or more programming languages, such as Java, C, C#, C++, Python, etc., but the present disclosure is not limited thereto. When processing unit 301 loads program 310 from storage unit 302, image enhancement module 311 and image analysis module 312 may be run, and image enhancement module 311 corresponds to steps S11 and S12, and image analysis module 312 corresponds to step S13. Therefore, when processing unit 301 loads program 310 from storage unit 302, steps S11-S13 may be executed.
[0078] In one embodiment, the process 310 may also include a model building module, although not described in Figure 3 During the execution of the model building module, the processing unit 301 trains the mask extraction model 102 and the image analysis model 106 using the aforementioned loss function 108 and the training data set.
[0079] Figure 4 FIG. 4 is a system block diagram of an image analysis system 40 according to an embodiment of the present disclosure. Figure 4 As shown, the system 40 includes an image enhancement circuit 401 and an image analysis circuit 402 coupled to each other.
[0080] The image enhancement circuit 401 and the image analysis circuit 402 may be implemented by one or more dedicated electronic circuits, such as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), and / or various logic circuits. The image enhancement circuit 401 is configured to perform steps S11 and S12, and the image analysis circuit 402 is configured to perform step S13.
[0081] The image analysis system provided by various embodiments of the present disclosure can not only avoid the phenomenon of over-enhancement of the light source area and overexposure of the light source when the brightness of the dark area of the night image is enhanced, but can also be applied to scenes with similar characteristics of the target object and the background, overcoming the problem of poor image analysis results caused by indiscriminate enhancement of the whole image in the prior art. In addition, in actual application, the trained model can be deployed on the edge device or terminal device for independent execution, which can not only achieve economic benefits, but also protect important corporate information.
[0082] The above paragraphs are described using multiple aspects. Obviously, the teachings of this article can be implemented in multiple ways, and any specific architecture or function disclosed in the examples is only a representative case. According to the teachings of this article, it should be understood in the art that each aspect disclosed in this article can be implemented independently, or two or more aspects can be combined to implement.
[0083] Although the present disclosure has been described above with reference to the embodiments, it is not intended to limit the present disclosure. Any person skilled in the art may make slight changes and modifications without departing from the spirit and scope of the present disclosure. Therefore, the scope of protection of the invention shall be determined by the scope defined in the attached claims.
Claims
1. An image analysis system, comprising a storage unit and a processing unit, wherein the processing unit loads a program from the storage unit to execute: The image enhancement module inputs the original image into a pre-trained mask extraction model, obtains a plurality of masks corresponding to the original image output by the mask extraction model, and uses the masks to perform a sequence of image enhancement operations on the original image to obtain an enhanced image, wherein each mask is a parameter array having the same resolution as the original image, and each element of the parameter array indicates how the corresponding pixel of the original image performs the image enhancement operation corresponding to the mask. ; as well as An image analysis module inputs the enhanced image into a pre-trained image analysis model to obtain an inference result output by the image analysis model; The mask extraction model and the image analysis model are trained using the same loss function.
2. The image analysis system according to claim 1, wherein the processing unit further performs: The model building module uses the loss function and the training data set to train the mask extraction model and the image analysis model, wherein the training data set includes a plurality of training data, each of which includes a training image and label data corresponding to the training image.
3. The image analysis system of claim 1, wherein the image enhancement operation comprises a denoising operation; and The mask includes a first mask used for performing the denoising operation.
4. The image analysis system of claim 1, wherein the image enhancement operation comprises a contrast adjustment operation; and The mask includes a second mask used for performing the contrast adjustment operation.
5. The image analysis system of claim 1, wherein the image enhancement operation comprises a brightness adjustment operation; and The mask includes a third mask used for performing the brightness adjustment operation.
6. The image analysis system of claim 1, wherein the image enhancement operation comprises a sharpening operation; and The masks include a fourth mask for performing the sharpening operation.
7. The image analysis system of claim 1, wherein the mask comprises a first mask, a second mask, a third mask, and a fourth mask, and the image enhancement operation comprises a denoising operation, a contrast adjustment operation, a brightness adjustment operation, and a sharpening operation; and The processing unit uses the first mask to perform the denoising operation on the original image to obtain a denoised image, uses the second mask and the third mask to perform the contrast adjustment operation and the brightness operation on the denoised image respectively to obtain an adjusted image, and uses the fourth mask to perform the sharpening operation on the adjusted image to obtain the enhanced image.
8. The image analysis system of claim 1, wherein the signal-to-noise ratio of the original image is lower than a specified signal-to-noise ratio threshold; The contrast of the original image is lower than a specified contrast threshold; The original image contains an overexposed area, and the brightness value of each pixel in the overexposed area is higher than a specified brightness threshold; and / or The sharpness of the original image is lower than a specified sharpness threshold.
9. An image analysis system, include: An image enhancement circuit is configured to input an original image into a pre-trained mask extraction model, obtain a plurality of masks output by the mask extraction model corresponding to the original image, and perform a sequence of image enhancement operations on the original image using the masks to obtain an enhanced image; An image analysis circuit, coupled to the image enhancement circuit, configured to input the enhanced image into a pre-trained image analysis model to obtain an inference result output by the image analysis model; The mask extraction model and the image analysis model are trained using the same loss function.
10. The image analysis system of claim 9, wherein the image enhancement operation comprises a denoising operation; and The mask includes a first mask used for performing the denoising operation.
11. The image analysis system of claim 9, wherein the image enhancement operation comprises a contrast adjustment operation; and The mask includes a second mask used for performing the contrast adjustment operation.
12. The image analysis system of claim 9, wherein the image enhancement operation comprises a brightness adjustment operation; and The mask includes a third mask used for performing the brightness adjustment operation.
13. The image analysis system of claim 9, wherein the image enhancement operation comprises a sharpening operation; and The masks include a fourth mask for performing the sharpening operation.
14. The image analysis system of claim 9, wherein the mask comprises a first mask, a second mask, a third mask, and a fourth mask, and the image enhancement operation comprises a denoising operation, a contrast adjustment operation, a brightness adjustment operation, and a sharpening operation; and The image enhancement circuit is configured to use the first mask to perform the denoising operation on the original image to obtain a denoised image, use the second mask and the third mask to perform the contrast adjustment operation and the brightness operation on the denoised image respectively to obtain an adjusted image, and use the fourth mask to perform the sharpening operation on the adjusted image to obtain the enhanced image.
15. An image analysis method, performed by an image analysis system, the method include: Inputting the original image into a pre-trained mask extraction model, and obtaining a plurality of masks corresponding to the original image output by the mask extraction model; Using the mask to perform a sequence of image enhancement operations on the original image to obtain an enhanced image; and Inputting the enhanced image into a pre-trained image analysis model to obtain an inference result output by the image analysis model; The mask extraction model and the image analysis model are trained using the same loss function.
16. The image analysis method according to claim 15, further comprising: include: The mask extraction model and the image analysis model are trained using the loss function and the training data set, wherein the training data set includes a plurality of training data, each of which includes a training image and label data corresponding to the training image.
17. The image analysis method of claim 15, wherein the image enhancement operation comprises a denoising operation; and The mask includes a first mask used for performing the denoising operation.
18. The image analysis method of claim 15, wherein the image enhancement operation comprises a contrast adjustment operation; and The mask includes a second mask used for performing the contrast adjustment operation.
19. The image analysis method of claim 15, wherein the image enhancement operation comprises a brightness adjustment operation; and The mask includes a third mask used for performing the brightness adjustment operation.
20. The image analysis method of claim 15, wherein the image enhancement operation comprises a sharpening operation; and The masks include a fourth mask for performing the sharpening operation.
21. The image analysis method according to claim 15, wherein the mask comprises a first mask, a second mask, a third mask and a fourth mask, the image enhancement operation comprises a denoising operation, a contrast adjustment operation, a brightness adjustment operation and a sharpening operation, and the step of performing the image enhancement operation on the original image using the mask to obtain the enhanced image further comprises: include: Using the first mask to perform the denoising operation on the original image to obtain a denoised image; Using the second mask and the third mask to perform the contrast adjustment operation and the brightness operation on the denoised image respectively to obtain an adjusted image; and The sharpening operation is performed on the adjusted image using the fourth mask to obtain the enhanced image.
Citation Information
Patent Citations
Image enhancement method, training method of image enhancement model and related equipment
CN114627034A
Image restoration method based on mask training
CN115115537A
Deep learning model training method, target detection method, device and equipment
CN116310356A
High Resolution Inpainting with a Machine-learned Augmentation Model and Texture Transfer
US20230342890A1
