Image enhancement apparatus and image enhancement method for processing region of interest
By detecting the region of interest in the image enhancement device and using a variational autoencoder to improve the image quality, the problems of large amount of calculation and inaccurate image quality in the prior art are solved, and efficient image enhancement effect is achieved.
Patent Information
- Application Number
- CN202411884191.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-19
- Publication Date
- 2025-04-15
AI Technical Summary
The prior art performs frame-by-pixel calculations on the entire image in monitoring applications, resulting in excessive calculation volume and inaccurate improvement of picture quality.
Design an image enhancement device, including a preprocessing circuit, an intelligent processor and a processor, to improve image quality by detecting areas of interest, using a variational autoencoder, and mixing the original and enhanced area data by mixing proportions, replacing the area of interest in the image.
Effectively improve the image quality of the area of interest, reduce the consumption of system resources, and improve the identification accuracy of monitoring applications.
Smart Images
Figure CN119831850A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image enhancement technology, and in particular to an image enhancement device and an image enhancement method for processing a region of interest. Background Art
[0002] In existing surveillance applications, when a camera captures an object to be detected, an image processing device usually enhances the quality of the entire image so that the user can see a clearer image to better identify the detected object. However, the above method involves frame-by-frame and pixel-by-pixel calculations, which results in an excessive amount of overall calculations and may also result in inaccurate image quality enhancement. Summary of the invention
[0003] In some embodiments, one of the objectives of the present application is to provide an image enhancement device and an image enhancement method that can improve the image quality of a region of interest in an image, so as to improve the deficiencies of the prior art.
[0004] In some embodiments, the image enhancement device includes a preprocessing circuit, an intelligent processor, and a processor. The preprocessing circuit preprocesses an input image data to generate a first image data. The intelligent processor detects at least one region of interest in the first image data to generate at least one original region data, and improves the image quality of the at least one original region data to generate at least one enhanced region data. The processor mixes the at least one original region data and the at least one enhanced region data according to a mixing ratio to generate at least one mixed region data, and replaces the at least one region of interest in the first image data with the at least one mixed region data to generate an output image data.
[0005] In some embodiments, an image enhancement method performed by an image enhancement device includes the following operations: preprocessing an input image data to generate a first image data; detecting at least one region of interest in the first image data to generate at least one original region data, and improving the image quality of the at least one original region data to generate at least one enhanced region data; and mixing the at least one original region data and the at least one enhanced region data according to a mixing ratio to generate at least one mixed region data, and replacing the at least one region of interest in the first image data with the at least one mixed region data to generate an output image data.
[0006] The features, implementation and effects of the present application are described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0007] Figure 1 A schematic diagram of an image enhancement device is drawn according to some embodiments of the present application;
[0008] Figure 2 According to some embodiments of the present application, Figure 1 A flowchart of the operation of the image intensifier;
[0009] Figure 3 To generate a drawing according to some embodiments of the present application Figure 1 A schematic diagram of an operation of at least one original region data;
[0010] Figure 4 To generate a drawing according to some embodiments of the present application Figure 1 A schematic diagram of the operation of at least one enhanced area data; and
[0011] Figure 5 An operation flow chart of an image enhancement method is drawn according to some embodiments of the present application.
[0012] 100: image enhancement device;
[0013] 101: Image sensor;
[0014] 110: preprocessing circuit;
[0015] 120: processor;
[0016] 130: Intelligent processor;
[0017] 140: memory;
[0018] 301: Bounding box;
[0019] 302: at least one region of interest;
[0020] 400: Variational Autoencoder;
[0021] 410: encoder;
[0022] 420: adder;
[0023] 430: residual convolution mapping module;
[0024] 440: decoder;
[0025] 500: Image enhancement methods;
[0026] BR: mixing ratio;
[0027] D1: image data;
[0028] D2: at least one original region data;
[0029] D3: at least one enhanced area data;
[0030] D4: at least one mixed area data;
[0031] DIN: input image data;
[0032] DO: output image data;
[0033] NL: total number of layers in the transition zone;
[0034] NS: Gaussian noise;
[0035] ROI_x: horizontal axis;
[0036] ROI_y: vertical coordinate;
[0037] ROI_w: width;
[0038] ROI_h: height;
[0039] S210, S220, S230, S240, S250: operation;
[0040] S510, S520, S530: operation;
[0041] SD1, SD2: at least one data. DETAILED DESCRIPTION
[0042] All words used in this application have their usual meanings. The definitions of the above words in commonly used dictionaries, including any use examples of words discussed herein in the content of this application are only examples and should not limit the scope and meaning of this application. Similarly, this application is not limited to the various embodiments shown in this specification.
[0043] As used in this application, "coupled" or "connected" may refer to two or more elements making direct physical or electrical contact with each other, or making indirect physical or electrical contact with each other, or may refer to two or more elements operating or acting on each other. As used in this application, the term "circuit" may refer to a device composed of at least one transistor and / or at least one active and passive element connected in a certain manner to process signals.
[0044] Figure 1 A schematic diagram of an image enhancement device 100 is drawn according to some embodiments of the present application. In some embodiments, the image enhancement device 100 can enhance the image quality of at least one region of interest (ROI) in image data to provide a more reliable monitoring application.
[0045] In some embodiments, the image enhancement device 100 may include a pre-processing circuit 110, a processor 120, an intelligence processing unit (IPU) 130, and a memory 140. The pre-processing circuit 110 may receive input image data D IN from an image sensor 101 (for example, but not limited to, a camera), and pre-process the input image data DIN to generate image data D1. In some embodiments, the pre-processing may include, but not limited to, image scaling, image color gamut conversion, etc. In some embodiments, the pre-processing circuit 110 may be implemented by an image processing circuit.
[0046] The processor 120 may store the image data D1 in the memory 140. The intelligent processor 130 may detect at least one region of interest in the image data D1 to generate at least one original region data D2, and enhance the image quality of at least one original region data D2 to generate at least one enhanced region data D3. In some embodiments, the processor 120 is a general-purpose processor that can perform a wide range of computing tasks; in contrast, the intelligent processor 130 is a dedicated processor that can be used for parallel processing of a large number of machine learning models, and is optimized for matrix operations and / or tensor operations, and is mainly used to perform tasks related to artificial intelligence models (and / or neural network models).
[0047] For example, the intelligent processor 130 may execute a neural network model for detecting at least one specific object (target object) to detect whether the at least one specific object exists in the image data D1. In some embodiments, the at least one specific object may include, but is not limited to, a face, a license plate, or at least one of the above two. The neural network model may be trained in advance with large-scale data and stored in the memory 140, so that the intelligent processor 130 can execute the neural network model by accessing the memory 140 to detect whether the specific object exists in the image data D1. If the at least one specific object is detected in the image data D1, the intelligent processor 130 may set at least one region of interest according to the position of the at least one specific object in the image data to generate at least one original region data D2. The operation of setting at least one region of interest will be described later. Figure 3 In some embodiments, the above neural network model can be implemented by a depthwise separable convolution neural network to reduce the amount of computation.
[0048] Next, the intelligent processor 130 may execute a variational autoencoder to process at least one original region data D2 to improve the image quality of at least one original region data D2 and generate at least one enhanced region data D3 accordingly. In some embodiments, the variational autoencoder may be implemented by a neural network model and may be stored in the memory 140 in advance, so that the intelligent processor 130 may execute the neural network model by accessing the memory 140. Related examples of variational autoencoders will be referred to later. Figure 4 In some embodiments, the memory 140 may be, but is not limited to, a dynamic random access memory.
[0049] The processor 120 may mix at least one original region data D2 and at least one enhanced region data D3 according to the mixing ratio BR to generate at least one mixed region data D4, and replace the aforementioned at least one region of interest with the at least one mixed region data to generate output image data DO. In other words, the processor 120 may mix the at least one enhanced region data D3 after the image quality enhancement with the at least one original region data D2 corresponding to the at least one region of interest and replace the corresponding data content of the at least one region of interest in the image data D1 accordingly. In this way, the image quality of at least one region of interest including the at least one specific object may be enhanced, and at the same time, the enhanced output image data DO may be prevented from being too unnatural. In some embodiments, the parameters of the mixing ratio BR may be stored in the memory 140.
[0050] Figure 2 According to some embodiments of the present application, Figure 1 In operation S210, the preprocessing circuit 110 preprocesses the input image data DIN to generate image data D1. In operation S220, the intelligent processor 130 detects at least one specific object of the input image data DIN and sets at least one region of interest according to the position of the at least one specific object to generate at least one original region data D2.
[0051] For explanation of operation S220, please refer to Figure 3 , Figure 3 To generate a drawing according to some embodiments of the present application Figure 1 As described above, the intelligent processor 130 can execute a pre-trained neural network model to detect whether there is at least one specific object in the image data D1. Figure 3As shown, if the intelligent processor 130 determines that there is a specific object (such as a license plate) in the image data D1, the intelligent processor 130 can output the coordinates of a bounding box 301 corresponding to the specific object, and its format is (x, y, w, h), where x and y are the horizontal coordinate (such as the coordinate in the horizontal direction) and the vertical coordinate (such as the coordinate in the vertical direction) of the center point of the specific object, and w and h are the width and height of the specific object, respectively.
[0052] Next, the intelligent processor 130 may convert the coordinate information of the bounding box 301 to set the aforementioned at least one region of interest, so that the at least one region of interest is a specific region that is sufficient to completely include the specific object and is located in the middle of the at least one region of interest. Figure 3 As shown, the intelligent processor 130 can set the coordinates of the upper left corner of at least one region of interest 302 to (ROI_x, ROI_y, ROI_w, ROI_h), where ROI_x and ROI_y are the horizontal coordinate and the vertical coordinate of the upper left corner of at least one region of interest 302, respectively, and ROI_w and ROI_h are the width and height of at least one region of interest 302, respectively. In some embodiments, the values of ROI_w and ROI_h can be adjusted according to the specific object to be detected. For example, if the specific object to be detected is a face, the values of ROI_w and ROI_h can be set to 128; or, if the specific object to be detected is a license plate, the values of ROI_w and ROI_h can be set to 256. The above values are for example only, and the present application is not limited thereto.
[0053] In some embodiments, the value of the horizontal coordinate ROI_x may be set to the minimum of a first value and a second value, the first value may be the difference between the horizontal coordinate x of the specific object and the half width of the specific object (e.g., w / 2), and the second value may be the difference between the horizontal coordinate x of the specific object and the half width of at least one region of interest 302 (e.g., ROI_w / 2). The above setting method may be described as the following mathematical formula: ROI_x=min(xw / 2, x-ROI_w / 2). Similarly, in some embodiments, the value of the vertical coordinate ROI_y may be set to the minimum of a third value and a fourth value, the third value may be the difference between the vertical coordinate y of the specific object and the half length of the specific object (e.g., h / 2), and the fourth value may be the difference between the vertical coordinate y of the specific object and the half width of at least one region of interest 302 (e.g., ROI_h / 2). The above setting method may be described as the following mathematical formula: ROI_y=min(yh / 2, y-ROI_h / 2). Through the above setting method, the intelligent processor 130 can make the specific object be located in the middle of the set at least one region of interest 302 as much as possible, and accordingly output the relevant information of the at least one region of interest 302 as at least one original region data D2.
[0054] The above operation is only explained by taking the example of setting a corresponding region of interest for a specific object. It should be understood that if there are more than one specific object in the image data D1 (for example, including several license plates, several faces, or more than one license plate and / or more than one face at the same time), the intelligent processor 130 can generate several regions of interest and corresponding several original area data through the aforementioned operation.
[0055] Continue to refer to Figure 2 In operation S230 , the intelligent processor 130 improves the image quality of at least one original region data D2 to generate at least one enhanced region data D3 .
[0056] For explanation of operation S230, please refer to Figure 4 , Figure 4 To generate a drawing according to some embodiments of the present application Figure 1 As described above, the intelligent processor 130 can execute the variational autoencoder to improve the image quality of at least one original regional data D2 to generate at least one enhanced regional data D3. Figure 4As shown, in some embodiments, the variational self-encoder 400 may include an encoder 410, an adder 420, a residual convolution mapping module 430, and a decoder 440. The intelligent processor 130 may input at least one original region data D2 to the encoder 410, so as to map the at least one original region data D2 in the pixel space to a feature vector (denoted as at least one data SD1) in the variable space through the encoder 410. The adder 420 may superimpose Gaussian noise NS on the at least one data SD1. The intelligent processor 130 may input the superposition result of the above two to the residual convolution mapping module 430, so as to generate an enhanced feature vector (denoted as at least one data SD2) through the residual convolution mapping module 430. Finally, the intelligent processor 130 may input the at least one data SD2 to the decoder 440, so as to map the at least one data SD2 back to at least one enhanced region data D3 in the pixel space through the decoder 440.
[0057] In some embodiments, each of the encoder 410, the residual convolution mapping module 430, and the decoder 440 can be implemented by a neural network model composed of several convolution layers, normalization layers, nonlinear activation functions, and the like, and executed by the intelligent processor 130. In some embodiments, since the encoder 410, the residual convolution mapping module 430, and the decoder 440 are responsible for different functions, for example, the encoder 410 is responsible for mapping the image before enhancement from the pixel space to the feature space, the residual convolution mapping module 430 is responsible for mapping the features before enhancement to the features after enhancement, and the decoder 440 is responsible for mapping the features after enhancement from the feature space back to the pixel space, the neural network models corresponding to the above three can be trained independently. In some embodiments, the training process of the neural network models corresponding to the above three can use the training mode of the generative adversarial network (GAN), so that these models can compete with each other, thereby obtaining a better enhanced image quality effect.
[0058] In some embodiments, the variational self-encoder 400 can add Gaussian noise NS to the image data to train the residual convolution mapping module 430 and the decoder 440 to reconstruct the corresponding image data with clear image quality when the image is affected by noise. In this way, the reliability and universality of the variational self-encoder 400 in enhancing image quality can be improved. In some embodiments, in actual applications, the residual convolution mapping module 430 (and / or the adder 420) can be automatically turned on under poor shooting environment conditions (for example, but not limited to, the shooting scene is dark or the shooting scene is rainy, etc.) to improve the quality of the captured image.
[0059] Continue to refer to Figure 2 In operation S240, the processor 120 mixes at least one original region data D2 and at least one enhanced region data D3 according to a mixing ratio BR to generate at least one mixed region data D4.
[0060] In some embodiments, the processor 120 determines a weight coefficient according to the total number of layers NL of the transition zone and the mixing ratio BR, and mixes at least one original region data D2 and at least one enhanced region data D3 according to the weight coefficient to generate at least one mixed region data D4. In some embodiments, the mixing ratio BR can be any value from 0 to 1 (including 0 and 1). In some embodiments, the transition zone can be the edge area of at least one region of interest, and the total number of layers NL of the transition zone represents the degree to which the transition zone is subdivided. In other words, the transition zone is a gradually changing area, and the edge of the enhanced region of interest image will gradually change to the edge of the original region of interest image in the transition zone. The larger the value of the total number of layers NL of the transition zone, the higher the degree of refinement of the transition zone, so that the change between each level is smaller, so that the image transition of the region of interest can be more natural. In some embodiments, the generation of at least one mixed region data D4 can be expressed as the following formula:
[0061] D4=WR×D3+(1-WR)×D4
[0062]
[0063] Wherein, the value w is the image width, the value h is the image height, the value WR is the aforementioned weight coefficient, the value i is the number of layers of the current transition zone, each of the values pw and ph is i+1, which can be used to indicate the pixel position of the current transition zone, and NL is the (total) number of layers of the transition zone. It can be seen from the above formula that when the number of layers i of the current transition zone increases, the weight coefficient WR will become smaller and smaller. Under this condition, the processor 120 will use more of at least one original area data D2 to mix the image at the corresponding pixel position. Alternatively, when the number of layers i of the current transition zone decreases, the weight coefficient WR will become larger and larger. Under this condition, the processor 120 will use more of at least one enhanced area data D3 to mix the image at the corresponding pixel position. That is, the weight coefficient WR gradually changes in the transition zone, so that the image content in the region of interest can show a smooth transition from the enhanced image to the original image. Explained in another way, by setting the transition zone and the above-mentioned weight coefficient WR, when mixing images, the processor 120 can use more of at least one original region data D2 in the pixel region close to the original image in the region of interest, and use more of at least one enhanced region data D3 in the pixel region far from the original image in the region of interest, thereby avoiding the appearance of an obvious boundary line at the edge of the image of the mixed region of interest. In some embodiments, the parameters of the total number of layers NL of the transition zone can be stored in the memory 140.
[0064] In operation S250, the processor 120 replaces the aforementioned at least one region of interest with at least one mixed region data D4 to generate output image data DO. After generating at least one mixed region data D4, the processor 120 may replace the image of at least one region of interest in the image data D1 with the image of at least one mixed region data D4, and generate output image data DO accordingly. In this way, the image of at least one region of interest in the output image data DO will be an enhanced image with better image quality, so that the user can view a clear surveillance image. In some embodiments, if the computing power provided by the hardware is sufficient, the above operation of the image enhancement device 100 can also be extended to real-time video streaming.
[0065] In some related technologies, when a specific object of interest is detected in an image, the image processing device will improve the image quality of the entire image. This method involves frame-by-frame and pixel-by-pixel calculations, so there are problems of large amount of calculation and inaccurate image quality improvement. Compared with the above technology, in some embodiments of the present application, when a specific object of interest is detected in an image, the image enhancement device 100 only processes the image content of the region of interest of the specific object, and uses a variational autoencoder with relatively less calculation to improve the image quality of the region of interest. In this way, the clarity of the specific object of interest in the output image can be effectively improved while saving system resources, so that subsequent monitoring and identification applications can more conveniently and accurately analyze the output image.
[0066] Figure 5 The following is a flowchart of an image enhancement method 500 according to some embodiments of the present application. In some embodiments, the image enhancement method 500 may be, but is not limited to, Figure 1 The image enhancement device 100 is implemented.
[0067] In operation S510, an input image data is preprocessed to generate a first image data. In operation S520, at least one region of interest in the first image data is detected to generate at least one original region data, and the image quality of the at least one original region data is improved to generate at least one enhanced region data. In operation S530, the at least one original region data and the at least one enhanced region data are mixed according to a mixing ratio to generate at least one mixed region data, and the at least one region of interest in the first image data is replaced with the at least one mixed region data to generate an output image data.
[0068] The description of the multiple operations of the above-mentioned image enhancement method 500 can refer to the above-mentioned multiple embodiments, so it will not be repeated here. The above-mentioned multiple operations are only examples and are not limited to being performed in the order in this example. Without violating the operation mode and scope of each embodiment of the present application, the various operations under the image enhancement method 500 can be appropriately increased, replaced, omitted or performed in a different order. Alternatively, one or more operations under the image enhancement method 500 can be performed simultaneously or partially simultaneously.
[0069] In summary, the image enhancement device and its image enhancement method provided in some embodiments of the present application can effectively increase the image quality of the area of interest while saving system resources, thereby improving the accuracy of subsequent monitoring applications (such as face recognition and vehicle recognition).
[0070] Although the embodiments of the present application are described above, these embodiments are not intended to limit the present application. Technicians in this technical field may change the technical features of the present application based on the explicit or implicit contents of the present application. All these changes may fall within the scope of patent protection sought by the present application. The scope of patent protection of the present application shall be based on the scope of the patent application of this specification.
Claims
1. An image enhancement device, characterized in that: include: A preprocessing circuit preprocesses an input image data to generate a first image data; an intelligent processor, detecting at least one region of interest in the first image data to generate at least one original region data, and improving the image quality of the at least one original region data to generate at least one enhanced region data; as well as A processor mixes the at least one original region data and the at least one enhanced region data according to a mixing ratio to generate at least one mixed region data, and replaces the at least one region of interest in the first image data with the at least one mixed region data to generate an output image data.
2. The image enhancement device according to claim 1, characterized in that: The intelligent processor executes a variational autoencoder to improve the image quality of the at least one original region data and generate the at least one enhanced region data.
3. The image enhancement device according to claim 2, characterized in that: The intelligent processor inputs the at least one original region data into an encoder of the variational autoencoder to generate at least one first data, inputs the at least one first data into a residual convolution mapping module of the variational autoencoder to generate at least one second data, and inputs the at least one second data into a decoder of the variational autoencoder to generate the at least one enhanced region data.
4. The image enhancement device according to claim 3, characterized in that: Before inputting the at least one first data into the residual convolution mapping module, the intelligent processor also uses an adder of the variational autoencoder to superimpose a Gaussian noise on the at least one first data.
5. The image enhancement device according to claim 1, characterized in that: The intelligent processor detects at least one target object in the first image data, and sets the at least one region of interest according to the position of the at least one target object to generate the at least one original region data.
6. The image enhancement device according to claim 5, characterized in that: The horizontal coordinate of the upper left corner coordinate of the at least one region of interest is set to the minimum of a first value and a second value, the first value is the difference between the horizontal coordinate of the at least one target object and half the width of the at least one target object, and the second value is the first value is the difference between the horizontal coordinate of the at least one target object and half the width of the at least one region of interest.
7. The image enhancement device according to claim 5, characterized in that: The vertical coordinate of the upper left corner coordinate of the at least one region of interest is set to the minimum of a first value and a second value, the first value being the difference between the longitudinal coordinate of the at least one target object and half the length of the at least one target object, and the second value being the first value being the difference between the longitudinal coordinate of the at least one target object and half the length of the at least one region of interest.
8. The image enhancement device according to claim 5, characterized in that: The at least one target object includes at least one of a face or a license plate.
9. The image enhancement device according to claim 1, characterized in that: The processor determines a weight coefficient according to a total number of layers of a transition zone and the mixing ratio, and mixes the at least one original region data and the at least one enhanced region data according to the weight coefficient to generate the at least one mixed region data.
10. The image enhancement device according to claim 9, characterized in that: The weight coefficient changes gradually in the transition zone.
11. An image enhancement method, characterized in that: Executed by an image enhancement device, the image enhancement method includes: Preprocessing an input image data to generate a first image data; Detecting at least one region of interest in the first image data to generate at least one original region data, and improving the image quality of the at least one original region data to generate at least one enhanced region data; and The at least one original region data and the at least one enhanced region data are mixed according to a mixing ratio to generate at least one mixed region data, and the at least one region of interest in the first image data is replaced by the at least one mixed region data to generate an output image data.