Image enhancement method, device and equipment
Through maximum and average pooling feature image processing, combined with image subtraction and normalization processing, the problem of difficulty in distinguishing noise information from effective information in existing technologies is solved, and the image enhancement effect and the performance of the visual task model are improved.
Patent Information
- Application Number
- CN202410291125.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-13
- Publication Date
- 2025-09-16
AI Technical Summary
In the image enhancement process, the existing technology has difficulty in accurately distinguishing noise information from valid information, resulting in poor enhancement effect. Especially when noise information and valid information overlap in the document image, the accuracy of the preset threshold is required to be high.
Through maximum pooling and average pooling feature image processing, noise information and high-frequency information are retained and reduced respectively. The pixel value in the maximum pooling feature image is replaced by the maximum value in the region, and the pixel value in the average pooling feature image is replaced by the average value in the region to reduce discrete high-frequency information. Finally, effective information is extracted through image subtraction and normalization.
It achieves the accurate distinction and extraction of effective information in the image without affecting the noise information, improves the image enhancement effect, and especially improves the learning ability of the visual task model in document images.
Smart Images
Figure CN120655512A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer vision, and in particular to an image enhancement method, apparatus, and device. Background Art
[0002] Computer vision refers to the ability of computer systems to acquire, process, and understand information through images. Computer vision tasks encompass a range of image analysis and processing tasks, including image classification, object recognition, object tracking, and object segmentation. Because images may contain noise, image enhancement is often required to enhance the effective information and improve the accuracy of computer vision tasks.
[0003] Currently, the following method can be used to enhance the effective information in an image: scan the image based on a preset threshold, determine that pixels in the image with pixel values greater than the preset threshold are noise information in the image, and then remove the noise information determined above to obtain the effective information in the image. It should be understood that in actual applications, there may be overlap between the noise information and the effective information in the image. For example, noise information such as shading, folds, and watermarks in an image that resemble symbols or text is often difficult to distinguish from the symbols or text in the image. This leads to the above method having a high requirement for the accuracy of the preset threshold. When the accuracy of the preset threshold is not high, the noise information determined using the above method may be inaccurate, resulting in poor image enhancement effect. Summary of the Invention
[0004] The present application provides an image enhancement method, apparatus and device, which can better enhance the effective information in the image.
[0005] In a first aspect, the present application provides an image enhancement method. The method can be applied to an image enhancement device. Specifically, the image enhancement device obtains an image to be processed, and then obtains a maximum pooling feature image based on the image to be processed, wherein the maximum pooling feature image includes a first region, and the image to be processed includes a second region, wherein the size of the first region is the same as the size of the second region, and the coordinates of the pixels in the first region in the maximum pooling feature image are the same as the coordinates of the pixels in the second region in the image to be processed, and the value of the pixel in the first region is the maximum value of the values of multiple pixels in the second region. Afterwards, the image enhancement device obtains an average pooling feature image based on the maximum pooling feature image, wherein the average pooling feature image includes a third region, and the maximum pooling feature image includes a fourth region, wherein the size of the third region is the same as the size of the fourth region, and the coordinates of the pixels in the third region in the average pooling feature image are the same as the coordinates of the pixels in the fourth region in the maximum pooling feature image, and the value of the pixels in the third region is the average value of the values of multiple pixels in the fourth region, and the size of the fourth region is larger than the size of the second region. Afterwards, the image enhancement device obtains a target image based on the image to be processed and the average pooling feature image.
[0006] In the technical solution provided by the present application, the image enhancement device can retain the noise information in the image to be processed and reduce the effective information in the image to be processed by replacing small pixel values in the image to be processed (for example, pixel values of valid information such as text) with large pixel values (for example, pixel values of noise information such as shading, folds, and watermarks). Therefore, the maximum pooling feature image retains the noise information in the image to be processed. For the maximum pooling feature image, the image enhancement device also reduces the discrete high-frequency information in the maximum pooling feature image by replacing the pixel values in the image with the average pixel values in the corresponding area. Therefore, the noise information contained in the average pooling feature image is more accurate. Therefore, the technical solution provided by the present application can accurately extract the noise information in the image to be processed, thereby achieving the purpose of image enhancement.
[0007] In one possible implementation, the image enhancement device obtains a maximum pooling feature image based on the image to be processed, including: the image enhancement device uses the maximum pooling kernel to scan the image to be processed, replaces the value of each pixel in the second area of the image to be processed with the maximum value of the multiple pixels in the area, and obtains the maximum pooling feature image.
[0008] In one possible implementation, the image enhancement device obtains an average pooling feature image based on the maximum pooling feature image, including: the image enhancement device uses the average pooling kernel to scan the maximum pooling feature image, replaces the value of each pixel in the fourth area of the maximum pooling feature image with the average value of the values of multiple pixels in the area, and obtains the average pooling feature image.
[0009] In one possible implementation, the size of the average pooling kernel is larger than the size of the maximum pooling kernel. By implementing the above implementation, the size of the fourth region can be made larger than the size of the second region, thereby achieving the purpose of reducing discrete high-frequency information in the maximum pooling feature image.
[0010] In one possible implementation, the image enhancement device obtains a target image based on the image to be processed and the average pooled feature image, including: the image enhancement device subtracts the values of pixels in the image to be processed from the values of pixels in the average pooled feature image to obtain a difference image, and then performs normalization processing on the difference image to obtain the target image.
[0011] It should be understood that since the average pooled feature image contains noise information extracted from the image to be processed, the above implementation method can retain the valid information in the image to be processed.
[0012] In a possible implementation, the image enhancement device obtains the image to be processed, including: the image enhancement device obtains the original image, and then performs grayscale processing on the original image to obtain the image to be processed. Here, the image to be processed is a grayscale image.
[0013] It should be understood that in practical applications, the original image acquired by the image enhancement device may be an RGB image. Compared with RGB images, grayscale images are easier to denoise, so implementing the above implementation method can further improve the image enhancement effect.
[0014] In one possible implementation, the original image is a document image. A document image is an image obtained by scanning or photographing a document (e.g., a bill, letter, contract, report, etc.). Document images often contain noise information such as shading, folds, and watermarks similar to symbols or text. Testing has found that using the technical solution provided in this application to perform image enhancement processing on document images can effectively improve the learning ability of the visual task model for document images.
[0015] In a second aspect, the present application provides an image enhancement device. The device may include an acquisition module, a pooling processing module, and a denoising module. The acquisition module is used to acquire the image to be processed. The pooling processing module is used to obtain a maximum pooling feature image based on the image to be processed, the maximum pooling feature image includes a first area, and the image to be processed includes a second area, wherein the size of the first area is the same as the size of the second area, and the coordinates of the pixels in the first area in the maximum pooling feature image are the same as the coordinates of the pixels in the second area in the image to be processed, and the value of the pixel in the first area is the maximum value of the values of multiple pixels in the second area. The pooling processing module is also used to obtain an average pooling feature image based on the maximum pooling feature image, the average pooling feature image includes a third area, and the maximum pooling feature image includes a fourth area, wherein the size of the third area is the same as the size of the fourth area, and the coordinates of the pixels in the third area in the average pooling feature image are the same as the coordinates of the pixels in the fourth area in the maximum pooling feature image, the value of the pixels in the third area is the average value of the values of multiple pixels in the fourth area, and the size of the fourth area is larger than the size of the second area. The denoising module is used to obtain a target image based on the image to be processed and the average pooling feature image.
[0016] In one possible implementation, the pooling processing module is used to scan the image to be processed using a maximum pooling kernel, and replace the value of each pixel in the second area of the image to be processed with the maximum value of the values of multiple pixels in the area, thereby obtaining a maximum pooling feature image.
[0017] In one possible implementation, the pooling processing module is used to scan the maximum pooling feature image using an average pooling kernel, and replace the value of each pixel in the fourth area of the maximum pooling feature image with the average value of multiple pixels in the area, thereby obtaining an average pooling feature image.
[0018] In one possible implementation, the size of the average pooling kernel is larger than the size of the maximum pooling kernel.
[0019] In one possible implementation, the denoising module is used to subtract the values of pixels in the image to be processed from the values of pixels in the average pooled feature image to obtain a difference image, and perform normalization processing on the difference image to obtain a target image.
[0020] In a possible implementation, the acquisition module is used to acquire an original image and perform grayscale processing on the original image to obtain an image to be processed.
[0021] In a possible implementation, the original image is a document image.
[0022] In a third aspect, the present application provides a computing device comprising a processor and a memory, wherein the processor is configured to execute instructions stored in the memory so that the computing device performs part or all of the method described in the first aspect and any implementation thereof.
[0023] In a fourth aspect, the present application provides a computing device cluster. The computing device cluster includes at least one computing device, each of which includes a processor and a memory. The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs part or all of the method described in the first aspect and any implementation thereof.
[0024] In a fifth aspect, the present application provides a computer program product comprising instructions. This computer program product may be software or a program product comprising instructions that can be run on a computing device or stored in any available medium. When this computer program product is run on a computing device, it causes the computing device to perform part or all of the method described in the first aspect and any implementation thereof.
[0025] In a sixth aspect, the present application provides a computer-readable storage medium comprising computer program instructions, which, when executed by a computing device, cause the computing device to perform part or all of the method described in the first aspect and any one of its implementations. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] Figure 1 is a schematic diagram of an image enhancement scenario provided by this application;
[0027] Figure 2 is a schematic diagram of an image enhancement method provided by this application;
[0028] Figures 3A to 3E This is an enhanced effect diagram of a bill image provided by this application;
[0029] Figure 4 Schematic diagram of a maximum pooling feature image provided by this application;
[0030] Figure 5 This is a schematic structural diagram of an image enhancement device provided by the present application;
[0031] Figure 6 It is a structural diagram of a computing device provided by this application;
[0032] Figure 7 This is a schematic diagram of the structure of a computing device cluster provided by this application;
[0033] Figure 8This is a structural diagram of another computing device cluster provided by this application. DETAILED DESCRIPTION
[0034] In order to solve the problem of poor image enhancement effect in the existing solutions, the present application provides an image enhancement method. The method can be applied to an image enhancement device. The image enhancement device can enhance the effective information in the image to be processed by performing a first pooling operation and a second pooling operation. Specifically, the image enhancement device can first perform a first pooling operation on the image to be processed to divide the image into multiple regions, and the values of all pixels in each region are replaced by the maximum value of the values of all pixels in the region, thereby reducing the effective information (for example, text) in each region and retaining the noise information (for example, shading, watermarks, folds, etc.) in each region. That is, the image obtained by the first pooling operation retains the noise information in the image to be processed. Afterwards, the image enhancement device performs a second pooling operation on the image obtained after the first pooling operation to divide the image into multiple regions, and the values of all pixels in each region are replaced by the average value of the values of all pixels in the region, thereby reducing the high-frequency information in each region, achieving the purpose of denoising the image obtained after the first pooling operation. That is, the image obtained after the second pooling operation includes more accurate noise information. Therefore, the technical solution provided by the present application can effectively distinguish between the valid information and the noise information in the image to be processed without affecting the noise information in the image to be processed, and can further more accurately extract the valid information in the image, thereby achieving the effect of image enhancement on the image to be processed.
[0035] The technical solution provided by this application is described below in conjunction with the accompanying drawings.
[0036] See also Figure 1 , Figure 1 Taking target recognition as an example, a schematic diagram of an image enhancement scenario applicable to this application is shown. It should be understood that in actual applications, the technical solution provided by this application can also be applied to other image enhancement scenarios, such as image classification, target tracking, target segmentation and other computer vision tasks, which are not specifically limited in this application. Figure 1 As shown, the scenario includes a client 100, an image enhancement device 200 and an image recognition device 300, wherein the client 100, the image enhancement device 200 and the image recognition device 300 are connected via a network, which may be a wide area network or a local area network.
[0037] The client 100 may be software or an application program deployed on a terminal device, such as a web client, wherein the terminal device may include, for example, a desktop computer, a laptop computer, a tablet computer, a smartphone, a wearable device, and the like.
[0038] The image enhancement device 200 can be a hardware system or a software system. When the image enhancement device 200 is a hardware system, the image enhancement device 200 can be implemented by a central processing unit (CPU), an application-specific integrated circuit (ASIC), or a programmable logic device (PLD). The PLD in this application can be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a data processing unit (DPU), a neural network processing unit (NPU), a system on chip (SoC), an offload card, an accelerator card, or any combination thereof. When the image enhancement device 200 is a software system, the image enhancement device 200 can be deployed on a single computing device or on a computing device cluster consisting of multiple computing devices, where the computing device can be a terminal server or a computing device in a data center (e.g., a server).
[0039] Similarly, the image recognition device 300 can also be a hardware system or a software system. When the image recognition device 300 is a hardware system, it can be implemented using a CPU, ASIC, or PLD. When the image recognition device 300 is a software system, it can be deployed on a single computing device or on a computing device cluster consisting of multiple computing devices.
[0040] exist Figure 1In the image enhancement scenario shown, the user can upload the image to be recognized (hereinafter referred to as the "original image") to the image enhancement device 200 through the client 100. After the image enhancement device 200 obtains the original image, it performs grayscale processing, a first pooling operation, a second pooling operation, an image subtraction process, and a normalization process on the original image in sequence, thereby obtaining an enhanced image (hereinafter referred to as the "target image"). Among them, the grayscale processing is used to convert the original image into a grayscale image, the first pooling operation is used to extract noise information in the grayscale image, the second pooling operation is used to denoise the above-extracted noise information, the image subtraction process is used to extract valid information in the grayscale image based on the denoised noise information, and the normalization process is used to scale the pixel values corresponding to the above-extracted valid information to a specific range. Afterwards, the image enhancement device 200 sends the target image to the image recognition device 300. The image recognition device 300 processes the target image based on the visual task model to obtain the content in the target image, thereby completing this image recognition.
[0041] Next, combine Figure 2 The flowchart of the data enhancement method shown in FIG. 1 describes in detail the process of image enhancement implemented by the image enhancement device 200 .
[0042] Step 101: The client 100 sends an original image to the image enhancement device 200. Accordingly, the image enhancement device 200 receives the original image sent by the client 100.
[0043] Specifically, the image enhancement device 200 provides an access interface, which can be implemented via an application programming interface (API) or a graphical user interface (GUI). The client 200 can send the original image to the image enhancement device 200 via the access interface. It should be understood that in addition to the above-described methods, the image enhancement device 200 can also obtain the original image via other methods, which are not limited in this embodiment.
[0044] As an example, the original image may be a document image. A document image is an image obtained by scanning or photographing a document (e.g., a bill, letter, contract, report, etc.). Document images often contain noise information such as shading, folds, and watermarks, similar to symbols or text.
[0045] Step 102: The image enhancement device 200 determines whether the original image is a grayscale image. If the original image is not a grayscale image, step 103 is executed; if the original image is a grayscale image, step 104 is executed.
[0046] A grayscale image is an image that contains brightness information but not color information. In a grayscale image, the value of a pixel (i.e., pixel value) represents the brightness of the pixel. Pixel values typically range from 0 to 255, where a pixel value of 0 indicates that the pixel is black, and a pixel value of 255 indicates that the pixel is white. When the pixel value is greater than 0 and less than 255, the larger the pixel value, the brighter the pixel.
[0047] Specifically, the image enhancement device 200 can determine whether the original image is a grayscale image based on the number of channels in the original image. If the number of channels in the original image is 1, the original image is a grayscale image; otherwise, the original image is not a grayscale image. It should be understood that in addition to the above method, the image enhancement device 200 can also use other methods to determine whether the original image is a grayscale image, which is not limited in this embodiment.
[0048] Step 103: The image enhancement device 200 performs grayscale processing on the original image to obtain a grayscale image.
[0049] Grayscale processing is an image processing method used to convert RGB images into grayscale images. An RGB image is an image that contains information about three color channels: red, green, and blue. A pixel in an RGB image is represented by three values, which correspond to the three color channels to indicate the intensity of the three colors in the pixel. Compared to RGB images, grayscale images are easier to denoise for the following reasons: RGB images include values for three color channels, and the relationship between the three color channels needs to be considered during the denoising process of RGB images, while grayscale images only contain brightness information, making them easier to denoise. In addition, the pixel values in grayscale images usually vary less, and the differences between adjacent pixels are not as significant as in RGB images. This makes the impact of noise information on grayscale images relatively small, making it easier to remove.
[0050] Specifically, the image enhancement device 200 can use formula (1) to process the pixels in the original image to obtain a grayscale image:
[0051] Y 1i =α×R i +β×G i +γ×B i (1)
[0052] In formula (1), R i , G i 、B i Respectively represent the red channel value, green channel value, and blue channel value of pixel i in the original image; α, β, and γ represent the weights of the red channel value, the green channel value, and the blue channel value, respectively. As an example, α = 0.229, β = 0.587, and γ = 0.114; Y 1iIt represents the grayscale value of pixel i after grayscale processing, that is, the value of pixel i in the grayscale image (ie, Y1).
[0053] Step 104: The image enhancement device 200 obtains a maximum pooling feature image according to the grayscale image.
[0054] Specifically, the image enhancement apparatus 200 performs a first pooling operation on the grayscale image to obtain a maximum pooling feature image, wherein the value of a pixel in the maximum pooling feature image is the maximum value among the values of multiple pixels in a region (hereinafter referred to as "region A") where the corresponding pixel in the grayscale image is located.
[0055] In some embodiments, the image enhancement device 200 may perform a first pooling operation on the grayscale image in the following manner: the image enhancement device 200 scans the grayscale image using a maximum pooling kernel, and during the scanning process, the values of all pixels in the scanned area are assigned to the maximum value among the values of multiple pixels in the area, as specifically described in formula (2), thereby obtaining a maximum pooling feature image. The maximum pooling kernel is a window for performing the first pooling operation, which can slide on the grayscale image at a certain step, and each time it slides, the values of all pixels in the window are assigned to the maximum value among the values of all pixels in the window.
[0056]
[0057] In formula (2), R represents the scanned area in the grayscale image Y1, R includes pixel k, Y 1k Indicates the value of pixel k, max k∈R (Y 1k ) represents the maximum value of all pixels in R (i.e., the maximum pixel value), Y 2i Represents the value of pixel i after the maximum pooling operation, that is, the value of pixel i in the maximum pooling feature image (i.e., Y2).
[0058] It should be understood that the larger the pixel value in a grayscale image, the higher its brightness, and the brightness of the effective information in the grayscale image (for example, text, patterns, etc.) is usually smaller than the brightness of the noise information (for example, watermarks, folds, etc.). Therefore, the pixel value of the effective information in the grayscale image is usually smaller than the pixel value of the noise information. In this step, by performing the first pooling operation on the grayscale image using the maximum pooling kernel, the pixels with large pixel values in the grayscale image can be retained, and the values of the pixels with small pixel values in the grayscale image can be replaced with large pixel values. Therefore, through this step, the noise information in the grayscale image can be retained and the effective information in the grayscale image can be reduced.
[0059] In some embodiments, the size of the maximum pooling kernel can be set by the user or dynamically adjusted by the image enhancement device 200 based on actual conditions. For example, the image enhancement device 200 can set the size of the maximum pooling kernel based on the resolution of the grayscale image. The lower the resolution of the grayscale image, the smaller the maximum pooling kernel size can be set. Conversely, the higher the resolution of the grayscale image, the larger the maximum pooling kernel size can be set. The reason is as follows: the lower the resolution of the grayscale image, the fewer pixels the grayscale image contains, and the denser the distribution of information in the grayscale image. Therefore, using a small-sized maximum pooling kernel can scan a lot of information in the grayscale image at one time. Moreover, compared with a large-sized maximum pooling kernel, a small-sized maximum pooling kernel loses less information during the first pooling operation, thereby retaining more noise information in the grayscale image. The higher the resolution of the grayscale image, the more pixels the grayscale image contains, and the sparser the distribution of information in the grayscale image. Therefore, using a large-sized maximum pooling kernel can scan more information in the image at one time, thereby more accurately retaining noise information in the grayscale image.
[0060] In order to facilitate understanding of this step, the following uses the image enhancement device 200 to Figure 3A The grayscale image shown in FIG1 is used as an example to perform the first pooling operation: the image enhancement device 200 can use a maximum pooling kernel of size 7*7. Figure 3A The grayscale image shown in the figure performs the first pooling operation to obtain Figure 3B The maximum pooling feature image shown. By comparison Figure 3A and Figure 3B It can be seen that the maximum pooling feature image removes the effective information in the grayscale image (i.e. Figure 3A The pixels with small pixel values, such as the text and seal in the bill), retain the noise information in the grayscale image (i.e. Figure 3B Pixels with large medium pixel values, for example, folds in bills).
[0061] In addition, it should be understood that in the process of performing the first pooling operation on the grayscale image using the maximum pooling kernel, the values of pixels in adjacent scanned areas may be the same or different, which leads to the possibility of discrete high-frequency information in the maximum pooling feature image. Here, discrete high-frequency information refers to the scanned area in the maximum pooling feature image where the pixel values change dramatically, for example, Figure 3B For ease of understanding, the following is combined with Figure 4 Further explanation of "discrete high-frequency information in the maximum pooling feature image": Figure 4 As shown, Figure 4 shows the values of the pixels that the maximum pooled image may include. Figure 4It can be seen that the pixels in the maximum pooling feature image obtained after the first pooling operation belong to 9 scanned areas (i.e., area 1 to area 9). Among these 9 scanned areas, the values of the pixels in area 1 to area 8 are all 255, and the values of the pixels in area 9 are all 0. Compared with areas 1 to 8, the values of the pixels in area 9 are much different from those in the other 8 areas. Therefore, the pixels in area 9 are Figure 4 The discrete high-frequency information in the maximum pooled feature image shown. For the maximum pooled feature image, the discrete high-frequency information in the image can be understood as noise information in the image. Therefore, the maximum pooled feature image needs to be denoised. The specific denoising method can be found in step 105 below.
[0062] Step 105: The image enhancement device 200 obtains an average pooling feature image according to the maximum pooling feature image.
[0063] Specifically, the image enhancement device 200 performs a second pooling operation on the maximum pooled feature image to obtain an average pooled feature image, wherein the value of a pixel in the average pooled feature image is the average value of the values of multiple pixels in the region where the corresponding pixel in the maximum pooled feature image is located (hereinafter referred to as "region B").
[0064] In some embodiments, the image enhancement device 200 can perform a second pooling operation on the maximum pooling feature image in the following manner: the image enhancement device 200 uses an average pooling kernel to scan the maximum pooling feature image, and during the scanning process, the values of all pixels in the scanned area are assigned to the average value of the values of multiple pixels in the area, as shown in formula (3), thereby obtaining an average pooling feature image. The average pooling kernel is a window for performing the second pooling operation, which can slide on the maximum pooling feature image with a certain step, and each time it slides, the values of all pixels in the window can be assigned to the average value of the values of all pixels in the window. In addition, it should be noted that the scanned area in this step is area B.
[0065]
[0066] In formula (3), n represents the number of pixels in the scanned area of the maximum pooling feature image Y2, Y 2k Indicates the value of pixel k in the scanned area, Y 3i Represents the value of pixel i after the average pooling operation, that is, the value of pixel i in the average pooling feature image (i.e., Y3).
[0067] In some embodiments, the size of the average pooling kernel can be set by the user or dynamically adjusted by the image enhancement device 200 according to actual conditions. However, it is worth noting that, regardless of the setting method, the size of the average pooling kernel should be larger than the size of the maximum pooling kernel for the following reasons: if the size of the average pooling kernel is larger than the size of the maximum pooling kernel, the size of the corresponding scanned area (i.e., area B) when the average pooling kernel scans the maximum pooling feature image will be larger than the size of the corresponding scanned area (i.e., area A) when the maximum pooling kernel scans the grayscale image, which allows area B to include not only pixels in area A, but also part or all of the pixels in area A adjacent to area A. When the adjacent area A is the area where discrete high-frequency information in the maximum pooling feature image is located, the discrete high-frequency information in area B can be removed by performing the second pooling operation on the pixels in area B. On the contrary, if the size of the average pooling kernel is smaller than or equal to the size of the maximum pooling kernel, the size of region B will be smaller than or equal to the size of region A, which makes the pixels in region B part or all of the pixels in region A. When region A is the region where discrete high-frequency information in the maximum pooling feature image is located, all pixels in region B are discrete high-frequency information. Then, by performing the second pooling operation on the pixels in region B, the values of the pixels in region B will not change, and thus the discrete high-frequency information in region B cannot be removed. In summary, the size of the average pooling kernel should be larger than the size of the maximum pooling kernel.
[0068] In order to facilitate understanding of this step, the following uses the image enhancement device 200 to Figure 3B The maximum pooling feature image shown in the figure performs the second pooling operation as an example for explanation: the image enhancement device 200 can use the average pooling kernel of size 21*21 Figure 3B The maximum pooling feature image shown in performs the second pooling operation, resulting in Figure 3C The average pooled feature image shown. By comparison Figure 3B and Figure 3C It can be seen that the average pooling feature image removes the discrete high-frequency information in the maximum pooling feature image (for example, Figure 3B black pixels in the .
[0069] Step 106: The image enhancement device 200 obtains a difference image based on the grayscale image and the average pooling feature image.
[0070] Specifically, the image enhancement device 200 performs an image subtraction operation on the grayscale image and the average pooled feature image to obtain a difference image. The image subtraction operation is used to subtract the values of pixels in the grayscale image from the values of pixels in the average pooled feature image. For details, see formula (4).
[0071] Y 4i=|Y 1i -Y 3i | (4)
[0072] In formula (4), Y 1i Represents the value of pixel i in the grayscale image Y1, Y 3i represents the value of pixel i in the average pooled feature image Y3, Y 4i The value of pixel i in the difference image (ie, Y4).
[0073] It should be understood that since the average pooling feature image removes the effective information in the grayscale image and retains the noise information in the grayscale image, and the above-mentioned grayscale processing, the first pooling operation and the second pooling operation do not involve changes in the image size, that is, the pixels in the grayscale image, the maximum pooling feature image and the average pooling feature image correspond one to one, so by performing an image subtraction operation on the grayscale image and the average pooling feature image, the noise information in the grayscale image can be removed and the effective information in the grayscale image can be obtained.
[0074] In order to facilitate the understanding of this step, the following Figure 3A The grayscale image shown and Figure 3C The image enhancement device 200 takes the average pooling feature image shown as an example to illustrate: Figure 3A The values of the pixels in the grayscale image shown and Figure 3C The corresponding pixel values in the average pooling feature image shown are subtracted to obtain Figure 3D The difference images shown. By comparison Figure 3A and Figure 3D It can be seen that Figure 3D The effective information (such as text, seal, etc.) in Figure 3C The effective information in is clearer.
[0075] Furthermore, the image enhancement device 200 may also perform normalization processing on the difference image, as specifically described in step 107. It should be understood that step 107 is an optional step, that is, the image enhancement device 200 may or may not perform this step, which is not limited in this embodiment.
[0076] Step 107: The image enhancement device 200 performs normalization processing on the difference image to obtain a target image.
[0077] Specifically, the image enhancement device 200 performs normalization processing on the difference image according to the minimum pixel value and the maximum pixel value in the difference image, as shown in formula (5), thereby obtaining the target image.
[0078]
[0079] In formula (5), Y4i represents the value of pixel i in the difference image Y4, max represents the maximum value of all pixels in the difference image (here specifically 255), min represents the minimum value of all pixels in the difference image (here specifically 0), Y 5i It represents the value of pixel i after normalization, that is, the value of pixel i in the target image (ie, Y5).
[0080] It should be understood that the normalization process can eliminate the scale differences between different features in the difference image, so that the values of all pixels in the target image are within a uniform range, which is helpful for learning subsequent visual task models.
[0081] In order to facilitate understanding of this step, the following uses the image enhancement device 200 to Figure 3D The normalization process of the difference image shown in FIG. 1 is performed as an example to illustrate: the image enhancement device 200 performs normalization process on the difference image shown in FIG. Figure 3D The difference image shown is normalized to obtain Figure 3E The target image shown is the enhanced image.
[0082] Step 108: The image enhancement device 200 sends the target image to the image recognition device 300. Accordingly, the image recognition device 300 receives the target image sent by the image enhancement device 200 and obtains the content of the target image according to the target image.
[0083] Specifically, the image recognition device 300 pre-stores a visual task model. This model can be an existing neural network model that is effective for target recognition, such as the Yolo (you only look once: unified) model, the SSD (single shot multi-box detector) model, or the RCNN (region convolutional neural network) model. After receiving a target image, the image recognition device 300 processes the target image based on the visual task model and identifies the content in the target image.
[0084] In the image enhancement method described in steps 101 to 108 above, the image enhancement processing of the original image by the image enhancement device 200 can be divided into multiple stages, and the operations corresponding to these multiple stages include at least two of grayscale processing, a first pooling operation, a second pooling operation, an image subtraction operation, and a normalization process. As can be seen from the above description of these operations, these operations can reduce the influence of noise information on the grayscale image corresponding to the original image, thereby improving the denoising effect of the grayscale image corresponding to the original image and achieving image enhancement of the original image. In addition, since the target image is obtained by performing the operations of the above multiple stages on the original image, wherein the operations corresponding to each stage can be understood as image changes that may be encountered in actual applications, the visual task model can learn richer and more diverse data in the process of processing the target image, thereby improving the performance and generalization ability of the visual task model and improving the accuracy of the visual task.
[0085] Combined with the above Figures 1 to 4 The image enhancement method provided by this application is introduced in detail. Figure 5 The structure of the image enhancement device 200 for executing the above-mentioned image enhancement method is described.
[0086] See also Figure 5 , Figure 5 FIG. 2 shows a schematic structural diagram of an image enhancement device 200. It should be understood that Figure 5 This is only an exemplary division method of the structure of the image enhancement device 200. In actual applications, the structure of the image enhancement device 200 may also have other division methods, which are not limited in this embodiment. Figure 5 As shown, the image enhancement device 200 includes an acquisition module 201 , a pooling processing module 202 and a denoising module 203 .
[0087] The acquisition module 201 is used to execute the step of receiving the original image in the above step 101, at least one of the above steps 102 and 103; the pooling processing module 202 is used to execute at least one of the above steps 104 and 105; the denoising module 203 is used to execute at least one of the above steps 106 and 107.
[0088] The acquisition module 201, pooling module 202, and denoising module 203 can all be implemented in software or hardware. Exemplarily, the implementation of acquisition module 201 is described using acquisition module 201 as an example. Similarly, the implementation of pooling module 202 and denoising module 203 can refer to the implementation of acquisition module 201.
[0089] As an example of a software functional unit, the acquisition module 201 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Furthermore, the computing instance may be one or more. For example, the acquisition module 201 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple geographically close data centers. Typically, a region may include multiple AZs.
[0090] Similarly, multiple hosts / virtual machines / containers running the code can be distributed within the same virtual private cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Cross-region communication between two VPCs within the same region, or between VPCs in different regions, requires a communication gateway within each VPC to interconnect the VPCs.
[0091] As an example of a hardware functional unit, the acquisition module 201 may include at least one computing device, such as a server, etc. Alternatively, the acquisition module 201 may also be a device implemented using a CPU, an ASIC, or a PLD, etc.
[0092] The multiple computing devices included in the acquisition module 201 can be distributed in the same region or in different regions. The multiple computing devices included in the acquisition module 201 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the acquisition module 201 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices included in the acquisition module 201 can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, GALs, DPUs, NPUs, SoCs, offload cards, and accelerator cards.
[0093] It should be noted that, in other embodiments, the acquisition module 201 can be used to perform any step in the above-described image enhancement method, the pooling module 202 can be used to perform any step in the above-described image enhancement method, and the denoising module 203 can be used to perform any step in the above-described image enhancement method. The steps that the acquisition module 201, the pooling module 202, and the denoising module 203 are responsible for implementing can be specified according to actual needs. The full functionality of the image enhancement device 200 is achieved by having the acquisition module 201, the pooling module 202, and the denoising module 203 respectively implement different steps in the above-described image enhancement method.
[0094] The present application also provides a computing device, which may be a terminal server or a computing device in a data center. Figure 6 The following is an exemplary diagram showing the structure of the computing device provided by this application. Figure 6 As shown, the computing device 400 includes a bus 401, a processor 402, a memory 403, and a communication interface 404, and the processor 402, the memory 403, and the communication interface 404 communicate with each other through the bus 401. It should be understood that the present application does not limit the number of processors 402 and memories 403 in the computing device 400. For simplicity, Figure 6 A processor 402 and a memory 403 are used as an example for description.
[0095] The bus 401 may be a peripheral component interconnect (PCI) bus, an extended industry standard architecture (EISA) bus, a unified bus (UBus or UB), a compute express link (CXL), a cache coherent interconnect for accelerators (CCIX), etc., wherein the unified bus is also called a Lingqu bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 4. Although only one line is used in the figure, this does not mean that there is only one bus or one type of bus in computing device 400. Bus 401 may include a path for transmitting information between various components of computing device 400 (eg, processor 402, memory 403, and communication interface 404).
[0096] The processor 402 may include any one or more computing units with computing capabilities, such as a CPU, a graphics processing unit (GPU), a microprocessor (MP) or a digital signal processor (DSP), an ASIC, an FPGA, a CPLD, an NPU, a SoC, an offload card, an accelerator card, etc.
[0097] Memory 403 may include volatile memory, such as random access memory (RAM). Memory 403 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD). In addition, memory 403 may also be implemented using storage class memory (SCM), phase change memory (PCM), or other types of storage media.
[0098] It is worth noting that the same type of storage medium can be configured in the same computing device to implement the function of memory 403, or two or more types of storage media can be configured to implement the function of memory 403. This application does not limit this.
[0099] Memory 403 stores executable program code. Processor 402 executes the executable program code to implement the functions of acquisition module 201, pooling module 202, and denoising module 203, thereby implementing the image enhancement method. In other words, memory 403 stores instructions for performing the image enhancement.
[0100] The communication interface 404 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to enable communication between the computing device 400 and other devices or a communication network. For example, the computing device 400 communicates with the client 100 and the image recognition device 300 through the communication interface 404 .
[0101] The present application also provides a computing device cluster, which includes multiple computing devices. The multiple computing devices in the computing device cluster may include terminal servers or computing devices in a data center, or may include terminal devices such as desktop computers, laptops, or smart phones. Figure 7The schematic diagram of the structure of the computing device cluster provided by this application is shown as an example. Figure 7 As shown, computing device cluster 500 includes multiple computing devices 400 .
[0102] The memory 403 of one or more computing devices 400 in the computing device cluster 500 may respectively store the same instructions for executing the above-mentioned image enhancement method. The memory 403 of one or more computing devices 400 in the computing device cluster 500 may also respectively store partial instructions for executing the above-mentioned image enhancement method. In other words, a combination of one or more computing devices 400 can jointly execute the above-mentioned image enhancement method.
[0103] It should be noted that the multiple computing devices 400 in the computing device cluster 500 can be connected via a network, which can be a wide area network or a local area network. Figure 8 , Figure 8 A possible implementation is shown. Figure 8 As shown, computing device 400A and computing device 400B are connected via a network. Specifically, the connection to the network is achieved through communication interfaces in computing device 400A and computing device 400B. In this implementation, memory 403 in computing device 400A may store instructions for executing the functions of acquisition module 201, and memory 403 in computing device 400B may store instructions for executing the functions of pooling module 202 and denoising module 203.
[0104] Figure 8 The connection method between the computing device cluster 500 shown may be based on the consideration that the image enhancement device 200 in the image enhancement method provided in the present application may be accessed by a large number of clients 100 at the same time, thereby receiving a large number of original images. Therefore, it is considered to hand over the functions implemented by the acquisition module 201 to the computing device 400A for execution.
[0105] It should be understood that Figure 8 The functions of the computing device 400A in FIG. 4 can also be completed by multiple computing devices 400 , and similarly, the functions of the computing device 400B can also be completed by multiple computing devices 400 .
[0106] The present application also provides a computer program product comprising instructions. The computer program product may be software or a program product comprising instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on a computing device, the computing device executes the image enhancement method described above.
[0107] The present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of being stored by a computing device, or a data storage device such as a data center that contains one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives). The computer-readable storage medium includes instructions that instruct the computing device to execute the image enhancement method described above.
[0108] Finally, it should be noted that the above embodiments are intended only to illustrate the technical solutions of this application and are not intended to limit them. Although this application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they may modify the technical solutions described in the above embodiments or replace some of the technical features therein with equivalents; and such modifications or replacements do not deviate from the scope of protection of the technical solutions of the embodiments of this application.
Claims
1. An image enhancement method, characterized in that: The method comprises: Get the image to be processed; Obtaining a maximum pooling feature image according to the image to be processed, the maximum pooling feature image including a first region, and the image to be processed including a second region, wherein a size of the first region is the same as a size of the second region, and coordinates of pixels in the first region in the maximum pooling feature image are the same as coordinates of pixels in the second region in the image to be processed, and a value of a pixel in the first region is a maximum value of values of multiple pixels in the second region; Obtaining an average pooling feature image according to the maximum pooling feature image, the average pooling feature image includes a third region, and the maximum pooling feature image includes a fourth region, wherein a size of the third region is the same as a size of the fourth region, and a coordinate of a pixel in the third region in the average pooling feature image is the same as a coordinate of a pixel in the fourth region in the maximum pooling feature image, a value of a pixel in the third region is an average of values of multiple pixels in the fourth region, and a size of the fourth region is larger than a size of the second region; A target image is obtained according to the image to be processed and the average pooled feature image.
2. The method according to claim 1, characterized in that The obtaining of a maximum pooling feature image according to the image to be processed includes: The image to be processed is scanned using the maximum pooling kernel, and the value of each pixel in the second area is replaced by the maximum value of multiple pixel values in the second area to obtain the maximum pooling feature image.
3. The method according to claim 2, characterized in that The obtaining the average pooling feature image according to the maximum pooling feature image includes: The maximum pooling feature image is scanned using an average pooling kernel, and the value of each pixel in the fourth region is replaced by an average value of the values of multiple pixels in the fourth region to obtain the average pooling feature image.
4. The method according to claim 3, characterized in that The size of the average pooling kernel is larger than the size of the maximum pooling kernel.
5. The method according to any one of claims 1 to 4, characterized in that The step of obtaining a target image according to the image to be processed and the average pooled feature image includes: Subtracting the values of the pixels in the image to be processed from the values of the pixels in the average pooled feature image to obtain a difference image; Normalization is performed on the difference image to obtain the target image.
6. The method according to any one of claims 1 to 5, characterized in that The step of obtaining an image to be processed includes: Get the original image; Grayscale processing is performed on the original image to obtain the image to be processed.
7. The method according to claim 6, characterized in that The original image is a document image.
8. An image enhancement device, characterized in that: The device comprises: An acquisition module, used for acquiring an image to be processed; A pooling processing module, configured to obtain a maximum pooling feature image according to the image to be processed, the maximum pooling feature image including a first region, the image to be processed including a second region, wherein the size of the first region is the same as the size of the second region, and the coordinates of the pixels in the first region in the maximum pooling feature image are the same as the coordinates of the pixels in the second region in the image to be processed, and the value of the pixel in the first region is the maximum value of the values of multiple pixels in the second region; and, obtaining an average pooling feature image according to the maximum pooling feature image, the average pooling feature image including a third region, the maximum pooling feature image including a fourth region, wherein the size of the third region is the same as the size of the fourth region, and the coordinates of the pixels in the third region in the average pooling feature image are the same as the coordinates of the pixels in the fourth region in the maximum pooling feature image, the value of the pixels in the third region is the average value of the values of multiple pixels in the fourth region, and the size of the fourth region is larger than the size of the second region; The denoising module is used to obtain a target image based on the image to be processed and the average pooled feature image.
9. The device according to claim 8, characterized in that The pooling processing module is used to scan the image to be processed using the maximum pooling kernel, replace the value of each pixel in the second area with the maximum value of multiple pixels in the second area, and obtain the maximum pooling feature image.
10. The device according to claim 9, characterized in that The pooling processing module is used to scan the maximum pooling feature image using an average pooling kernel, set the value of each pixel in the fourth area to the average value of multiple pixels in the fourth area, and obtain the average pooling feature image.
11. The device according to claim 10, characterized in that The size of the average pooling kernel is larger than the size of the maximum pooling kernel.
12. The device according to any one of claims 8 to 11, characterized in that The denoising module is configured to subtract corresponding pixel values in the image to be processed from pixel values in the average pooled feature image to obtain a difference image; and perform normalization processing on the difference image to obtain the target image.
13. The device according to any one of claims 8 to 12, characterized in that The acquisition module is used to acquire the original image and perform grayscale processing on the original image to obtain the image to be processed.
14. The device according to claim 13, characterized in that The original image is a document image.
15. A computing device, characterized in that The method comprises a processor and a memory, wherein the processor is configured to execute instructions stored in the memory so as to enable the computing device to perform the operation steps of the method according to any one of claims 1 to 7.
16. A computing device cluster, characterized in that: comprising at least one computing device, each computing device including a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in a memory of the at least one computing device, so that the computing device cluster performs the operation steps of the method according to any one of claims 1 to 7.
17. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster is caused to perform the operation steps of the method according to any one of claims 1 to 7.
18. A computer-readable storage medium, characterized in that The method comprises computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster performs the operation steps of the method according to any one of claims 1 to 7.