Image processing method and related apparatus

Abstract

The embodiment of the application discloses an image processing method, the method comprises the following steps: after an original image is acquired by a data processing device, the original image is down-sampled, a deep network is calculated to generate a depth bilateral grid, an original affine transformation matrix in the depth bilateral grid is corrected to obtain a new affine transformation matrix, and the data processing device performs affine filtering according to the new affine transformation matrix to generate an enhanced image. The enhanced image obtained according to the corrected new affine transformation matrix effectively reduces the negative optimization phenomenon.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more particularly to an image processing method and related apparatus. Background Technology

[0002] The existing massive video and image resources are limited by factors such as complex shooting environments, device pixel response performance, unprofessional shooting techniques, and network transmission compression distortion. These factors result in problems such as uneven brightness distribution, insufficient layering, inadequate saturation, and even blurry or lost details, which seriously affect the user's viewing experience.

[0003] Image enhancement can effectively improve the above problems. Image enhancement is a way to purposefully emphasize the overall or local characteristics of an image, making the original image clearer or emphasizing certain features that are of interest to the user.

[0004] Existing image enhancement methods involve downsampling the original image to generate a depth-based bilateral mesh, and then upsampling the depth-based bilateral mesh to generate an original affine transformation matrix. The image obtained by performing affine filtering based on this affine transformation matrix may exhibit negative optimization in certain scenarios, such as overexposure of highlights, noise in shadows, and unnatural skin tones. Summary of the Invention

[0005] The first aspect of this application provides an image processing method, including:

[0006] After acquiring the original image, the data processing device downsamples the original image, generates a depth bilateral mesh through deep network calculation, and obtains a new affine transformation matrix by correcting the original affine transformation matrix in the depth bilateral mesh. The data processing device then performs affine filtering based on the new affine transformation matrix to generate the enhanced image.

[0007] The enhanced image obtained from the corrected new affine transformation matrix effectively reduces the phenomenon of negative optimization.

[0008] Based on the first aspect of the embodiments of this application, in the first embodiment of the first aspect of the embodiments of this application, the specific process of the data processing device generating a new affine transformation matrix can be as follows: the data processing device upsamples the depth bilateral grid to generate the original affine transformation matrix, and corrects the original affine transformation matrix to obtain a new affine transformation matrix.

[0009] This application provides a specific implementation method for generating a new affine transformation matrix.

[0010] Based on the first aspect or the first implementation of the first aspect of the present application, in the second implementation of the first aspect of the present application, the data processing device determines a highlight suppression curve, which indicates the weight of the original image and the network-enhanced image when they are fused at different gray values. The data processing device can correct the original affine transformation matrix according to the highlight suppression curve.

[0011] In this application, a method is provided to correct the original affine transformation matrix by using a highlight suppression curve. The image generated by the corrected affine transformation matrix effectively reduces the probability of highlight overexposure.

[0012] Based on the second implementation of the first aspect of the present application, in the third implementation of the first aspect of the present application, the function corresponding to the highlight suppression curve is a continuous function, and the function corresponding to the highlight suppression curve is monotonically differentiable.

[0013] Based on any one of the first to third embodiments of the present application, in the fourth embodiment of the first aspect of the present application, the data processing device determines a memory color region, and the enhancement of the memory color region is relatively low compared to other regions of the image.

[0014] In this embodiment of the application, a method for distinguishing memory color regions is provided.

[0015] Based on any one of the first to fourth embodiments of the present application, in the fifth embodiment of the first aspect of the present application, when the resolution of the original image is greater than a preset threshold, the data processing device performs multiple downsampling on the original image and generates a depth bilateral grid after calculation by a depth network.

[0016] In this embodiment of the application, the data processing device can downsample the original image multiple times to ensure the efficiency of image processing.

[0017] A second aspect of this application provides a data processing apparatus that performs the methods described in the first aspect and any embodiment thereof.

[0018] A third aspect of this application provides a computer storage medium storing instructions that, when executed on a computer, cause the computer to perform the methods described in the first aspect and any embodiment of the first aspect.

[0019] The fourth aspect of this application provides a computer software product that, when executed on a computer, causes the computer to perform the methods described in the first aspect and any implementation thereof. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating an image processing method in an embodiment of this application;

[0021] Figure 2 This is a schematic diagram of the bilateral mesh principle in an embodiment of this application;

[0022] Figure 3 This is another flowchart illustrating the image processing method in an embodiment of this application;

[0023] Figure 4 This is another flowchart illustrating the image processing method in an embodiment of this application;

[0024] Figure 5 This is a schematic diagram of deep network training and inference in the embodiments of this application;

[0025] Figure 6 This is a schematic diagram illustrating the fusion method for suppressing highlight regions in this application embodiment;

[0026] Figure 7 These are schematic diagrams illustrating several possible highlight suppression curves in the embodiments of this application;

[0027] Figure 8 This is a schematic diagram of the skin tone preservation process in an embodiment of this application;

[0028] Figure 9 This is a schematic diagram of the memory color retention curve in an embodiment of this application;

[0029] Figure 10 This is a schematic diagram of a data processing device in an embodiment of this application;

[0030] Figure 11 This is another structural schematic diagram of the data processing device in the embodiments of this application. Detailed Implementation

[0031] The technical solutions in the embodiments of this application will be described below with reference to the accompanying drawings. In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, or B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the related objects before and after are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, and c can be single or multiple. It is worth noting that "at least one" can also be interpreted as "one or more".

[0032] It should be noted that, in this application, the terms "exemplary" or "for example" are used to indicate that something is being described as an example, illustration, or illustration. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0033] The descriptions of "first," "second," etc., appearing in the embodiments of this application are for illustrative purposes and to distinguish the objects being described. They have no order and do not indicate any special limitation on the number of devices in the embodiments of this application, nor do they constitute any limitation on the embodiments of this application.

[0034] This application provides an image processing method for image enhancement and improving image quality.

[0035] In this embodiment, the data processing device can be a terminal device, also known as a user equipment (UE), mobile station (MS), mobile terminal (MT), etc., which refers to a device that provides voice and / or data connectivity to a user. Examples include handheld devices with wireless connectivity and in-vehicle devices. Currently, some examples of terminals include: mobile phones, tablets, laptops, PDAs, mobile internet devices (MIDs), wearable devices, virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, and wireless terminals in smart homes.

[0036] See Figure 1 A flowchart of an image processing method in this application includes:

[0037] 101. The data processing equipment acquires the original image;

[0038] Data processing equipment acquires raw images. When the input is a video stream (such as online video, local video, screen mirroring, video call, etc.), the video stream can be processed by a video hardware decoder to obtain image frames, which are then used as the raw images.

[0039] 102. The data processing equipment downsamples the original image to generate a depth-based bilateral grid;

[0040] like Figure 2 As shown, a bilateral grid is essentially a 3D data structure that can store edge information. Assuming the resolution of the original image is 1024*1024, the original image can be downsampled to obtain a thumbnail. The reduction ratio of the thumbnail is not limited. For example, if the thumbnail resolution is 256*256, it means that a sample is selected from the original image to obtain an image reduced by 16 times. Other ratios are also possible, such as a thumbnail resolution of 512*512, which means a reduction of four times.

[0041] The data processing device feeds the thumbnail into a deep neural network to obtain a deep bilateral grid, such as... Figure 3 As shown, due to the enormous computational demands of deep neural networks, the original image is typically downsampled to obtain a thumbnail, which is then followed up to obtain a double-sided grid. Referring to Figure 4, if the resolution of the original image exceeds a preset threshold, such as when the original image is an ultra-high-resolution photograph (4K, 8K resolution, etc.), the data processing device can downsample the original image multiple times to ensure image processing efficiency.

[0042] 103. The data processing equipment upsamples the depth-bounded mesh to generate the original affine transformation matrix;

[0043] See Figure 5 During the training phase of a deep network, training samples are provided, such as original images, paired labeled images, and semantic masks of key objects. By controlling the loss function, the network fits the bilateral grid and semantic mask information. At this time, the deep bilateral grid generates a coefficient matrix through bilateral guided upsampling (BGU), which is the original affine transformation matrix. As shown below, it is an example of the original affine transformation matrix.

[0044]

[0045] 104. The data processing equipment corrects the original affine transformation matrix to obtain a new affine transformation matrix;

[0046] In this embodiment, the original affine transformation matrix can be modified to achieve effects such as highlight suppression, memory color preservation, and shadow preservation. These are explained below:

[0047] 1. Enhance highlights and shadows;

[0048] See Figure 6 , Figure 6 This diagram illustrates how a data processing device can fuse raw input and inference results to achieve highlight suppression. By adjusting the fusion intensity curves at different brightness values, the final enhancement effect on highlights and shadows can be controlled. During implementation, weight parameters for corresponding brightness values ​​can be pre-calculated to form a mapping lookup table for rapid adjustment of affine coefficients. (See also...) Figure 7 This diagram illustrates several possible highlight suppression curves S, where the horizontal axis X represents grayscale values ​​and the vertical axis Y represents the weights assigned to the original image and the network-enhanced image during fusion. The fusion curve typically satisfies the following properties: 1) continuous, differentiable, and monotonic; 2) a rapidly decreasing range and an easily adjustable slope; and 3) easily adjustable upper and lower limits (range) of fusion intensity.

[0049] 2. Memory color retention.

[0050] In an image, the hues of certain objects are memory colors, which differ significantly from the enhancement strategies for other color systems. Examples include the blue of the sky, the green of vegetation, and the skin tone of a person. Data processing devices identify memory color regions, and the enhancement of memory color regions is relatively low compared to other areas of the image.

[0051] Taking human skin tone as an example, on the one hand, different ethnicities have significant differences in skin tone, making it difficult to define the range of skin tone values; on the other hand, similar skin tones may also appear in the background area, making it impossible to effectively distinguish them simply by selecting color values. It's important to note that processing human skin tone tends to prioritize realism and avoids drastic enhancement. This means there's no need to significantly adjust the brightness, saturation, or contrast of the skin tone. When blending colors within a specific range, the original input should account for the majority of the weight; simultaneously, the weight parameters should be applied according to the mask information to mask non-human areas. (See also...) Figure 8 This is a diagram illustrating the process of skin tone retention. (See attached image.) Figure 9 A schematic diagram of curve H for preserving a memory color (such as skin tone).

[0052] When the original affine transformation matrix A is: Pixels are denoted as I = [RGB 1] T Where R, G, and B represent the red, green, and blue depths of the pixel, respectively.

[0053] Let W be the grayscale guide image of the original image, S be the highlight suppression curve, H be the memory color (such as skin color) preservation curve, and M be the semantic mask. The semantic mask M can be a matrix that marks each pixel of the image. Taking skin color as an example, the pixels of the skin part can be marked as 1, and the pixels of other parts can be marked as 0.

[0054] The result of the superimposed highlight suppression is: Y′=S(W(I))×A×I+(1-S(W(I)))×I;

[0055] The result of superimposing the memory color is: Y=(M×H(I))×Y′+(1-M×H(I))×Y′;

[0056] The corrected affine transformation matrix is:

[0057]

[0058] 105. The data processing equipment performs affine filtering based on the new affine transformation matrix to generate the enhanced image.

[0059] The data processing device performs affine filtering based on the new affine transformation matrix to generate an enhanced image, specifically X′=A′×I.

[0060] Combination Figure 5In this embodiment, during the forward inference stage of the network, in addition to using a bilateral grid, a highlight suppression parameter and a memory color preservation parameter combined with a semantic mask are added to adjust the coefficients of the original affine transformation matrix, forming a new affine transformation matrix with multiple functions, which is used for the final pixel value optimization calculation.

[0061] The data transmission method in the embodiments of this application has been described above. The apparatus in the embodiments of this application is described below. Please refer to [link / reference]. Figure 10 One embodiment of the data processing device in this application includes:

[0062] Acquisition unit 1001 is used to acquire the original image.

[0063] The generation unit 1002 is used to downsample the original image, generate a depth bilateral grid after calculation by a depth network, and perform affine filtering based on the new affine transformation matrix to generate an enhanced image.

[0064] The correction unit 1003 is used to correct the original affine transformation matrix in the depth bilateral mesh to obtain a new affine transformation matrix.

[0065] In this embodiment, the operations performed by each unit are the same as before. Figure 1 In the illustrated embodiments, the operations performed by the data processing device are described similarly and will not be repeated here.

[0066] Figure 11 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application. The data processing device 1100 may include one or more processors 1101 and a memory 1105, in which one or more applications or data are stored.

[0067] The memory 1105 can be volatile or persistent storage. The program stored in the memory 1105 may include one or more modules, each module including a series of instruction operations on the data processing device 1100. Furthermore, the processor 1101 may be configured to communicate with the memory 1105 and execute the series of instruction operations in the memory 1105 on the data processing device 1100.

[0068] The data processing device 1100 may also include one or more power supplies 1102, one or more wired or wireless network interfaces 1103, one or more input / output interfaces 1104, and / or one or more operating systems, such as any one of Microsoft Windows, Android, Mac OS, Unix, and Linux.

[0069] The processor 1101 can perform the operations performed by the data processing device in any of the foregoing embodiments, which will not be described in detail here.

[0070] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0071] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0072] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0073] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0074] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. An image processing method, characterized in that, include: Data processing equipment acquires raw images; The data processing device downsamples the original image and generates a depth-based bilateral mesh after calculation using a depth network. The data processing device corrects the original affine transformation matrix in the depth-bounded mesh to obtain a new affine transformation matrix; The data processing device performs affine filtering based on the new affine transformation matrix to generate an enhanced image; The method further includes: The data processing device determines a highlight suppression curve, which indicates the weight of the original image and the network-enhanced image when they are fused at different gray values. The data processing device corrects the original affine transformation matrix based on the highlight suppression curve.

2. The method according to claim 1, characterized in that, The data processing device corrects the original affine transformation matrix in the depth-bounded mesh to obtain a new affine transformation matrix, including: The data processing device upsamples the depth-bounded mesh to generate the original affine transformation matrix. The data processing device corrects the original affine transformation matrix to obtain a new affine transformation matrix.

3. The method according to claim 1, characterized in that, The function corresponding to the highlight suppression curve is a continuous function, and the function corresponding to the highlight suppression curve is monotonically differentiable.

4. The method according to any one of claims 1 to 3, characterized in that, The method further includes: The data processing device determines a memory color region, which has a lower enhancement level compared to other regions of the image.

5. The method according to any one of claims 1 to 3, characterized in that, The data processing device downsamples the original image and generates a depth-based bilateral mesh after calculation using a depth network, including: When the resolution of the original image is greater than a preset threshold, the data processing device downsamples the original image multiple times and generates a depth bilateral mesh after calculation by a depth network.

6. A data processing device, characterized in that, include: The acquisition unit is used to acquire the original image; The generation unit is used to downsample the original image and generate a depth-based bilateral mesh after calculation by a deep network; The correction unit is used to correct the original affine transformation matrix in the depth bilateral mesh to obtain a new affine transformation matrix; The generation unit is also used to perform affine filtering based on the new affine transformation matrix to generate an enhanced image; The data processing device further includes a determination unit for determining a highlight suppression curve, wherein the highlight suppression curve indicates the weight of the original image and the network-enhanced image when they are fused at different gray values; The correction unit corrects the original affine transformation matrix according to the highlight suppression curve.

7. The device according to claim 6, characterized in that, The correction unit is specifically used to upsample the depth bilateral mesh to generate the original affine transformation matrix, and to correct the original affine transformation matrix to obtain a new affine transformation matrix.

8. The device according to claim 6, characterized in that, The function corresponding to the highlight suppression curve is a continuous function, and the function corresponding to the highlight suppression curve is monotonically differentiable.

9. The device according to any one of claims 6 to 8, characterized in that, The determining unit is further configured to determine a memory color region, wherein the enhancement level of the memory color region is lower than that of other regions of the image.

10. The device according to any one of claims 6 to 8, characterized in that, The generation unit is specifically used to downsample the original image multiple times when the resolution of the original image is greater than a preset threshold, and generate a depth bilateral mesh after calculation by a depth network.

11. A computer storage medium, characterized in that, The computer storage medium stores instructions that, when executed on the computer, cause the computer to perform the method as described in any one of claims 1 to 5.

12. A computer program product, characterized in that, When the computer program product is executed on a computer, it causes the computer to perform the method as described in any one of claims 1 to 5.

Images