Image processing method and related system

Abstract

Embodiments of the present application provide an image processing method and related system. The method comprises: processing an initial image to obtain a semantic interest region map of the initial image, N enhanced processed images and N fusion weight coefficient maps; and obtaining a fused enhanced image according to the semantic interest region map, the N fusion weight coefficient maps and the N enhanced processed images. Through processing the initial image, the semantic interest region map of the initial image, the N enhanced processed images and the N fusion weight coefficient maps are obtained. Then, the semantic interest region map, the N enhanced processed images and the N fusion weight coefficient maps are fused to obtain the fused enhanced image. By combining the semantic interest region information and the enhanced processed image information, the ROI region prior information is obtained in the image enhancement process, which helps to improve the visual task accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to an image processing method and related system. BACKGROUND

[0002] Traditional image enhancement schemes usually include multiple sub-modules, each of which independently completes its own image processing subtask. For example, color correction, gamma correction, sharpening, color enhancement, etc. These sub-modules usually adopt a serial superposition manner to form an image processing pipeline for image processing, and finally obtain an enhanced image. Such an image processing procedure requires extremely high manpower and time cost for joint debugging of the coordination effect between modules to obtain optimal image quality and visual effect.

[0003] Deep learning methods, especially those based on convolutional neural networks (CNN), are the key driving force for the development of the field of artificial intelligence in recent years, and have achieved remarkable results in various tasks of computer vision. In the field of image restoration and image quality enhancement, deep learning-based methods have already surpassed traditional methods.

[0004] Both traditional methods and deep learning methods can obtain image quality that is friendly to human visual perception; however, such enhancement methods oriented to human visual perception are not necessarily friendly to visual tasks (such as object detection tasks), and are usually not friendly. For example, an image refined by an image expert will lose some targets in the scene, i.e., the targets cannot be normally detected; and an enhanced image obtained by a visual task-guided optimization image enhancement method, although the targets in the scene can be normally detected, has a low dynamic range and color loss. Both of these two ways will reduce the accuracy of visual tasks. SUMMARY

[0005] The present application discloses an image processing method and related system, which helps to improve the performance of visual tasks and improve the accuracy of visual tasks.

[0006] In a first aspect, an image processing method is provided, including: processing an initial image to obtain a semantic interest region map of the initial image, N enhanced images, and N fusion weight coefficient maps, N being an integer not less than 2; and obtaining a fused enhanced image according to the semantic interest region map, the N fusion weight coefficient maps, and the N enhanced images.

[0007] The embodiment of the application obtains a semantic interest region map of an initial image, N enhanced images and N fusion weight coefficient maps by processing the initial image, and then performs fusion processing on the semantic interest region map, the N enhanced images and the N fusion weight coefficient maps to obtain a fused enhanced image. By combining ROI semantic information and enhanced image information, the ROI region prior information is obtained in the image enhancement process, which helps to improve the visual task accuracy.

[0008] In a possible implementation, the obtaining of the fused enhanced image according to the semantic interest region map, the N fusion weight coefficient maps and the N enhanced images includes: obtaining N semantic fusion weight coefficient maps according to the semantic interest region map and the N fusion weight coefficient maps, the N semantic fusion weight coefficient maps corresponding to the N fusion weight coefficient maps; and performing fusion processing on the N enhanced images according to the N semantic fusion weight coefficient maps to obtain the fused enhanced image.

[0009] The present scheme focuses on the enhancement of the ROI region and weakens the non-ROI region, which is beneficial to the early perception of the features with the ROI optimization direction in the backend CV task. Moreover, the multi-dimensional image enhancement interacts with each other in the fusion process, and the information is communicated, so that more rich feature information can be obtained in the image enhancement process.

[0010] In a possible implementation, the method further includes: performing smoothing processing on the N semantic fusion weight coefficient maps respectively to obtain N smoothed semantic fusion weight coefficient maps; and the fusion processing on the N enhanced images according to the N semantic fusion weight coefficient maps to obtain the fused enhanced image includes: performing fusion processing on the N enhanced images according to the N smoothed semantic fusion weight coefficient maps to obtain the fused enhanced image.

[0011] The embodiment of the application performs image enhancement processing on an initial image in parallel, each processing task unit is independent and runs in parallel to complete a unique image processing task, and then the processing result is adaptively fused with a semantic interest region to obtain a fused enhanced image. The present scheme can obtain the ROI information of the visual task in the image enhancement process by introducing the ROI semantic map, and performs foreground and background enhancement before the enhancement process; and the image fusion processing based on semantic and model double perception design combines the ROI semantic information and the gradient of the backend visual task to obtain the ROI region prior information and the visual task model prior in the image enhancement process, which significantly improves the visual task accuracy.

[0012] In another possible implementation manner, the obtaining the fused enhanced image according to the semantic interest region map, the N fusion weight coefficient maps and the N enhanced processed images comprises: obtaining a fused initial enhanced image according to the N fusion weight coefficient maps and the N enhanced processed images; and obtaining the fused enhanced image according to the fused initial enhanced image and the semantic interest region map.

[0013] In a possible implementation manner, the processing the initial image to obtain the N enhanced processed images and the N fusion weight coefficient maps comprises: performing N times of enhancement processing on the initial image in parallel to obtain the N enhanced processed images and the N fusion weight coefficient maps, the N enhanced processed images corresponding to the N times of enhancement processing, and the N fusion weight coefficient maps corresponding to the N times of enhancement processing.

[0014] In this scheme, each enhancement unit is independently responsible for a respective enhancement task, and is more likely to obtain an enhancement effect of a dedicated feature. Meanwhile, the separate image processing units are easier to construct, do not need to be cooperatively debugged between units, and reduce the difficulty of finding an optimal solution of image enhancement. In addition, the error accumulation of the cascade serial mode is significantly reduced, and the running time is greatly reduced due to the parallel operation of the enhancement units.

[0015] In a second aspect, an embodiment of the present application provides an image processing apparatus, comprising: a processing module configured to process an initial image to obtain a semantic interest region map of the initial image, N enhanced processed images and N fusion weight coefficient maps, N being an integer greater than or equal to 2; and a fusion module configured to obtain a fused enhanced image according to the semantic interest region map, the N fusion weight coefficient maps and the N enhanced processed images.

[0016] In the embodiment of the present application, the initial image is processed to obtain a semantic interest region map of the initial image, N enhanced processed images and N fusion weight coefficient maps. Then, the semantic interest region map, the N enhanced processed images and the N fusion weight coefficient maps are fused to obtain a fused enhanced image. By combining the ROI semantic information and the image information after enhancement processing, the ROI region prior information is obtained in the image enhancement process, which helps to improve the visual task accuracy.

[0017] In a possible implementation manner, the fusion module is configured to: obtain N semantic fusion weight coefficient maps according to the semantic interest region map and the N fusion weight coefficient maps, the N semantic fusion weight coefficient maps corresponding to the N fusion weight coefficient maps; and perform fusion processing on the N enhanced processed images according to the N semantic fusion weight coefficient maps to obtain the fused enhanced image.

[0018] The scheme focuses on the enhancement of the ROI region and weakens the non-ROI region, which is beneficial to the early perception of the characteristics with the ROI optimization direction by the backend CV task. Moreover, the multi-dimensional image enhancement interacts with each other in the fusion process, and the information is communicated, so that more rich feature information can be obtained in the image enhancement process.

[0019] In a possible implementation, the processing module is further configured to perform smoothing processing on the N semantic fusion weight coefficient maps respectively to obtain N smoothed semantic fusion weight coefficient maps; and the fusion module is further configured to perform fusion processing on the N enhanced images according to the N smoothed semantic fusion weight coefficient maps to obtain the fused enhanced image.

[0020] In the embodiment of the present application, parallel image enhancement processing is performed on the initial image, each processing task unit is independent and runs in parallel, and unique image processing tasks are completed, and then the processing results are adaptively fused with the semantic interest region to obtain a fused enhanced image. In the scheme, by introducing the ROI semantic map, the ROI information of the visual task can be obtained in advance in the image enhancement process, and the foreground and background are enhanced before the enhancement process; and through the image fusion processing based on semantic and model double perception design, the ROI semantic information and the backend visual task gradient are combined, so that the image enhancement process obtains the ROI region prior information and the visual task model prior, and the visual task accuracy is significantly improved.

[0021] In another possible implementation, the fusion module is configured to obtain a fused initial enhanced image according to the N fusion weight coefficient maps and the N enhanced images, and obtain the fused enhanced image according to the fused initial enhanced image and the semantic interest region map.

[0022] In a possible implementation, the processing module is configured to perform N times of enhancement processing on the initial image in parallel to obtain N enhanced images and N fusion weight coefficient maps, the N enhanced images correspond to the N times of enhancement processing, and the N fusion weight coefficient maps correspond to the N times of enhancement processing.

[0023] In the scheme, each enhancement unit independently completes its own enhancement task, and the exclusive enhancement effect of the special feature is more easily obtained. At the same time, the separate image processing unit is easier to build, and does not need the cooperation and debugging between units, which reduces the difficulty of finding the optimal solution of image enhancement, significantly reduces the error accumulation of the cascade serial mode, and each enhancement unit runs in parallel, which greatly reduces the running time.

[0024] In a third aspect, an embodiment of the present application provides an image processing apparatus, comprising a processor and a memory; wherein the memory is configured to store program code, and the processor is configured to invoke the program code to execute the method provided in any of the implementation manners of the first aspect.

[0025] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method provided in any of the implementation manners of the first aspect.

[0026] In a fifth aspect, an embodiment of the present application provides a computer program product, characterized in that, when the computer program product is run on a computer, the computer is caused to execute the method provided in any of the implementation manners of the first aspect.

[0027] In a sixth aspect, an embodiment of the present application provides a chip system, which is applied to an electronic device; the chip system comprises one or more interface circuits and one or more processors; the interface circuit and the processor are interconnected through a line; the interface circuit is configured to receive a signal from a memory of the electronic device and send the signal to the processor, and the signal comprises computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device executes the method provided in any of the implementation manners of the first aspect.

[0028] It can be understood that the apparatus provided in the second aspect, the apparatus provided in the third aspect, the computer readable storage medium provided in the fourth aspect, the computer program product provided in the fifth aspect or the chip system provided in the sixth aspect are all used to execute the method provided in any of the first aspect. Therefore, the beneficial effects that can be achieved are referred to the beneficial effects in the corresponding method, which will not be described here. BRIEF DESCRIPTION OF DRAWINGS

[0029] The drawings used in the embodiments of the present application are described below.

[0030] Figure 1 is an architecture schematic diagram of an image processing system provided by an embodiment of the present application;

[0031] Figure 2 is a flow schematic diagram of an image processing method provided by an embodiment of the present application;

[0032] Figure 3 is a flow schematic diagram of another image processing method provided by an embodiment of the present application;

[0033] Figure 4 is an image processing schematic diagram for a human-vehicle target detection task provided by an embodiment of the present application;

[0034] Figure 5 This is a schematic diagram of a fusion process provided in an embodiment of this application;

[0035] Figure 6 This is a schematic flowchart of another image processing method provided in the embodiments of this application;

[0036] Figure 7 This is a schematic diagram of the structure of an image processing device provided in an embodiment of this application;

[0037] Figure 8 This is a schematic diagram of another image processing device provided in an embodiment of this application. Detailed Implementation

[0038] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0039] First, let's explain the terms used in this plan:

[0040] Image (image quality) enhancement: refers to techniques that process at least one of the following aspects of an image: brightness, color, contrast, saturation, dynamic range, etc., to meet certain specific indicators.

[0041] Artifacts refer to various forms of information that did not exist in the original image but appeared in the image after processing.

[0042] Computer vision (CV) tasks refer to the automatic extraction, analysis, and understanding of useful information from a single image or a series of images. These typically include fundamental tasks such as image classification, image instance segmentation, and image object detection.

[0043] Region of Interest (ROI): In machine vision and image processing, the region that needs to be processed from the image is delineated using shapes such as rectangles, circles, ellipses, and irregular polygons. This region of interest is called the ROI.

[0044] The area under the PR curve (Average Precision, AP) is a mainstream evaluation metric for object detection models. A higher AP value indicates higher model accuracy on the target class table.

[0045] Mean Average Precision (mAP): The average AP of the detected target across all categories. A higher mAP value indicates higher model accuracy and better performance.

[0046] It should be noted that the present application can be applied to image quality enhancement or video quality enhancement in various application scenarios such as automatic driving, video monitoring, smart city, and mobile phone intelligence. The present scheme can also be applied to tasks that need to meet the requirements of both visual task accuracy performance improvement and running speed.

[0047] The execution subject of the present scheme can be a mobile terminal, a video monitoring device, a vehicle-mounted device, or the like. By deploying on the related device, the effect of the image or video quality enhancement task is improved through software modification, so as to improve the performance (accuracy) of the visual task algorithm deployed in the backend.

[0048] Referring to Figure 1 Fig. 1 shows an architecture schematic diagram of an image processing system provided by an embodiment of the present application. As shown in Figure 1 The system can include an image preprocessing unit 1001, a multi-task parallel image enhancement unit 1002, a semantic and model perception fusion unit 1003, and a computing visual algorithm unit 1004.

[0049] The image preprocessing unit 1001 is configured to receive an input initial image, for example, a low-quality image, and output a preprocessed image by preprocessing the input initial image, so as to realize basic changes of the image to adapt to specific visual tasks. The preprocessing can be, for example, Gamma correction, color gamut conversion, and the like.

[0050] The multi-task parallel image enhancement unit 1002 is configured to receive the preprocessed image and output an enhanced image group (for example, N enhanced images) and a fusion weight coefficient group (for example, N fusion weight coefficient maps). The multi-task parallel image enhancement unit 1002 can include a plurality of sub-enhancement units, for example, a first image enhancement unit, a second image enhancement unit, and an Nth image enhancement unit. Each sub-enhancement unit completes an independent enhancement function, for example, High Dynamic Range Imaging (HDR) enhancement, contrast enhancement, sharpness enhancement, color enhancement, brightness enhancement, and the like. The specific types of sub-units can be selected and configured by a user or the like according to actual conditions, and the present scheme does not make specific limitations.

[0051] The semantic and model perception fusion unit 1003 is configured to receive the enhanced image group and the fusion weight coefficient group, and output a fused enhanced image. The semantic and model perception fusion unit 1003 includes a semantic ROI generation unit and a model perception fusion unit. The semantic ROI generation unit can be a pre-trained neural network configured to receive an input initial image and generate an ROI semantic map. The ROI semantic map indicates a region of interest for a back-end visual task. The model perception fusion unit is configured to receive the enhanced image group, the fusion weight coefficient group, and the ROI semantic map, and fuse to output a final enhanced image.

[0052] The computing visual algorithm unit 1004 is configured to receive the enhanced image and obtain a visual task target output. For example, the computing visual algorithm unit 1004 detects a target frame (target detection) or a segmentation region (instance segmentation). During the training process, the loss function gradient is transmitted to the image preprocessing unit 1001, the multi-task parallel image enhancement unit 1002, and the semantic and model perception fusion unit 1003 for network parameter update learning.

[0053] It should be noted that the image preprocessing unit 1001 and the computing visual algorithm unit 1004 are not necessarily required, and the present application does not make a specific limitation in this regard.

[0054] The present application introduces an ROI semantic map, so that the ROI region of the visual task can be obtained in advance during the image enhancement process, and the foreground and background are enhanced differently before the enhancement process. Through the image fusion processing based on semantic and model dual perception design, the ROI semantic information and the back-end visual task gradient are combined, so that the ROI region prior information and the visual task model prior are obtained during the image enhancement process, and the visual task accuracy is significantly improved.

[0055] Referring to Figure 2 FIG. 1 shows a flowchart of an image processing method according to an embodiment of the present application. As shown in Figure 2 The method includes steps 201-202, which are specifically as follows:

[0056] 201. Process an initial image to obtain a semantic region of interest map of the initial image, N enhanced images, and N fusion weight coefficient maps, where N is an integer greater than or equal to 2.

[0057] For example, a server processes an initial image.

[0058] The processing can include extracting a region of interest from the initial image to obtain a semantic region of interest map of the initial image.

[0059] The processing also includes performing enhancement processing on the initial image to obtain N enhanced images and N fusion weight coefficient maps. For example, the enhancement processing can be N different enhancement processing.

[0060] The enhanced image described above can be, for example, an HDR enhanced image, a contrast enhanced image, a sharpness enhanced image, a color enhanced image, a brightness enhanced image, and the like, and the present solution does not make specific limitations thereto.

[0061] 202. Obtain a fused enhanced image based on the semantic interest region map, the N fusion weight coefficient maps, and the N enhanced images.

[0062] Based on the semantic interest region map, the N fusion weight coefficient maps, and the N enhanced images obtained above, perform fusion processing to further obtain a fused enhanced image.

[0063] In the embodiments of the present application, the initial image is processed to obtain a semantic interest region map of the initial image, N enhanced images, and N fusion weight coefficient maps. Then, the semantic interest region map, the N enhanced images, and the N fusion weight coefficient maps are fused to obtain a fused enhanced image. By combining ROI semantic information and enhanced image information, the ROI region prior information is obtained in the image enhancement process, which helps to improve the visual task accuracy.

[0064] Referring to Figure 3 FIG. 3 shows a flowchart of another image processing method provided by the embodiments of the present application. As shown in Figure 3 The method includes steps 301-304, which are specifically as follows:

[0065] 301. Process an initial image to obtain a semantic interest region map of the initial image.

[0066] As a possible implementation manner, the processing of the initial image to obtain the semantic interest region map of the initial image can be that the initial image is input into a neural network for processing to further obtain the semantic interest region map of the initial image.

[0067] Of course, other manners can also be used to obtain the semantic interest region map, and the present solution does not make specific limitations thereto.

[0068] 302. Perform N times of enhancement processing on the initial image in parallel to obtain N enhanced images and N fusion weight coefficient maps, the N enhanced images corresponding to the N times of enhancement processing, and the N fusion weight coefficient maps corresponding to the N times of enhancement processing, N being an integer not less than 2.

[0069] The N times of enhancement processing on the initial image in parallel can be understood as N times of enhancement processing on the initial image respectively, and each time of enhancement processing is independent. The parallel processing can be simultaneous, and the present solution does not make a specific limitation on this.

[0070] That is to say, each time of enhancement processing respectively obtains an enhanced image and a fusion weight coefficient map. Based on the above-mentioned N times of parallel enhancement processing, N enhanced images and N fusion weight coefficient maps can be obtained.

[0071] The above-mentioned enhancement processing can be, for example, HDR enhancement processing, contrast enhancement processing, sharpness enhancement processing, color enhancement processing, brightness enhancement processing, etc., and the present solution does not make a specific limitation on this.

[0072] It should be noted that the order of steps 301 and 302 can be simultaneous execution, or can be in a predetermined order, etc., and the present solution does not make a specific limitation on this.

[0073] In the present solution, each enhancement unit independently takes charge of its own enhancement task, and it is easier to obtain the enhancement effect of exclusive characteristics. At the same time, the separate image processing unit is easier to build, does not need the cooperative commissioning between units, reduces the difficulty of finding the optimal solution of image enhancement, significantly reduces the error accumulation of the cascade serial mode, and each enhancement unit runs in parallel, greatly reduces the running time.

[0074] In one possible implementation manner, as shown in Figure 4 The image enhancement of the embodiment is applied to a human-vehicle target detection task. The initial image I input is preprocessed by Gamma correction to obtain a processed image P. The image P is subjected to parallel enhancement process to implement HDR, contrast, and sharpness enhancement three enhancement units. The image P is simultaneously subjected to the three enhancement units in parallel, and the corresponding enhanced image and fusion weight coefficient map are output.

[0075] 303. Obtain N semantic fusion weight coefficient maps according to the semantic interest region map and the N fusion weight coefficient maps, the N semantic fusion weight coefficient maps corresponding to the N fusion weight coefficient maps;

[0076] In one possible implementation manner, based on the obtained semantic interest region map and the N fusion weight coefficient maps, a point multiplication operation is performed to obtain N fusion weight coefficient maps with ROI semantic information.

[0077] In another possible implementation manner, based on the obtained semantic interest region map and the N fusion weight coefficient maps, a channel dimension pixel superposition is performed, and then a convolution operation is performed to obtain N fusion weight coefficient maps with ROI semantic information.

[0078] 304. Based on the N semantic fusion weight coefficient maps, perform fusion processing on the N enhanced images to obtain the fused enhanced image.

[0079] The N enhanced images are subjected to pixel multiplication and summation operations along the channel dimension based on the semantic fusion weight coefficient map to generate the final fused enhanced image.

[0080] like Figure 4 As shown, the semantic interest region map of the initial image obtained from the initial image I, along with three semantic fusion weight coefficient maps and three enhanced images, are fused to obtain the fused enhanced image O. Target detection is then performed based on this fused enhanced image O.

[0081] In one possible implementation, prior to step 304, the following may also be included:

[0082] The N semantic fusion weight coefficient maps are smoothed to obtain N smoothed semantic fusion weight coefficient maps.

[0083] Accordingly, step 304 can specifically be:

[0084] The N enhanced images are fused based on the N smoothed semantic fusion weight coefficient maps to obtain the fused enhanced image.

[0085] The smoothing process described above can be, for example, performing convolution on the N semantic fusion weight coefficient graphs to eliminate the obvious boundaries caused by the dot product, making the final semantic fusion weight coefficient graph smoother.

[0086] like Figure 5 The diagram illustrates a fusion processing method provided in this embodiment. An initial image is input into a semantic ROI generation unit (model) to obtain a semantic region of interest map. The initial image is then input into a multi-task parallel image enhancement unit (model) for processing to obtain a fusion weight coefficient map set and an enhanced image set. Next, the semantic region of interest map and the fusion weight coefficient map set are multiplied by pixels to obtain a fusion weight coefficient map set with ROI semantic information. This fusion weight coefficient map set with ROI semantic information is then convolved to eliminate some boundary effects, making it smoother. Finally, a channel-dimensional pixel multiplication and summation operation is performed on the obtained fusion weight coefficient map set with ROI semantic information and the enhanced image set to generate the final fused enhanced image.

[0087] In a possible implementation, the target frame and the like can be obtained by inputting the fused enhanced image into a backend target detection algorithm.

[0088] It should be noted that, in the training process, the system transmits corresponding loss function gradients for updating the parameters of the front-end image enhancement network.

[0089] In the embodiment of the present application, the initial image is processed in parallel, each processing task unit is independent and runs in parallel, and a unique image processing task is completed. Then, the processing result is combined with the semantic interest region for adaptive fusion to obtain a fused enhanced image. In this scheme, by introducing an ROI semantic map, the ROI information of the visual task can be obtained in advance in the image enhancement process, and the foreground and background are enhanced differently before the enhancement process. Through the image fusion processing based on semantic and model dual perception design, the ROI semantic information and the backend visual task gradient are combined to enable the image enhancement process to obtain the ROI region prior information and the visual task model prior, thereby significantly improving the visual task accuracy. The final fused enhanced image obtained in this way not only undergoes the above-mentioned parallel multiple image enhancement processing, but also fuses the ROI semantic map, so that the fused enhanced image can obtain the region that the visual task will focus on in advance in the image enhancement process, and the background is enhanced differently before the enhancement process. At the same time, the image is enhanced by, for example, HDR enhancement processing, contrast enhancement processing, sharpness enhancement processing, color enhancement processing, and brightness enhancement processing.

[0090] Referring to Figure 6 FIG. 2 shows a flowchart of another image processing method provided by the embodiment of the present application. As shown in Figure 6 The method includes steps 601-604, and the details are as follows.

[0091] 601, processing an initial image to obtain a semantic interest region map of the initial image;

[0092] As a possible implementation, the processing of the initial image to obtain the semantic interest region map of the initial image can be performed by inputting the initial image into a neural network for processing to obtain the semantic interest region map of the initial image.

[0093] Of course, other ways can also be used to obtain the semantic interest region map, which is not limited in the present application.

[0094] 602, performing N times of enhancement processing on the initial image in parallel to obtain N enhanced images and N fusion weight coefficient maps, the N enhanced images corresponding to the N times of enhancement processing, and the N fusion weight coefficient maps corresponding to the N times of enhancement processing, N being an integer not less than 2;

[0095] The N times of enhancement processing on the initial image in parallel can be understood as that the initial image is subjected to N times of enhancement processing respectively, and each time of enhancement processing is independent. The parallel processing can be simultaneous, and the present solution does not make a specific limitation thereon.

[0096] That is to say, each time of enhancement processing respectively obtains an enhanced image and a fusion weight coefficient map. Based on the parallel N times of enhancement processing, N enhanced images and N fusion weight coefficient maps can be obtained.

[0097] The enhancement processing can be, for example, HDR enhancement processing, contrast enhancement processing, sharpness enhancement processing, color enhancement processing, brightness enhancement processing, etc., and the present solution does not make a specific limitation thereon.

[0098] In a possible implementation manner, the respective enhancement processing corresponds to a preset model. In the training process, the respective model has a corresponding ground truth image and an output enhanced image to solve a loss value, and then network parameter update learning is performed, so that each model obtains unique and differentiated enhancement effect.

[0099] 603. obtaining a fused initial enhanced image according to the N fusion weight coefficient maps and the N enhanced images;

[0100] The N fusion weight coefficient maps and the N enhanced images are fused to obtain a fused initial enhanced image.

[0101] The fusion processing can be, for example, a channel dimension pixel point multiplication and summation operation.

[0102] 604. obtaining the fused enhanced image according to the fused initial enhanced image and the semantic interest region map.

[0103] The fused initial enhanced image obtained above and the semantic interest region map obtained above are subjected to a channel dimension point multiplication operation or a convolution operation after channel dimension superposition, and then a fused enhanced image is obtained.

[0104] In the embodiments of the present application, the initial image is subjected to image enhancement processing in parallel, each processing task unit is independent and runs in parallel to complete a unique image processing task, and then the processing result is adaptively fused with a semantic interest region to obtain a fused enhanced image. By using this method, the visual task detection accuracy can be significantly improved.

[0105] On the basis of the foregoing embodiments, the evaluation results of the image processing method provided by the embodiments of the present application on the data set BDD100K are shown in Table 1. The 10,000 sets of test data in the open source data set BDD100K are used for testing, and the data covers a wide range of scenes, including typical scenes such as daytime, nighttime, dusk, overcast, foggy, and rainy. The evaluation results are obtained by comparing a plurality of target detection algorithms in recent years, and the YOLO v4 is the target detection algorithm used in the embodiments.

[0106] Table 1

[0107]

[0108]

[0109] As shown in the sixth row of Table 1, the experimental results of the present scheme when only optimizing the image enhancement method while the parameters of the target detection algorithm are fixed, the mAP is improved by 2.46%. As shown in the seventh row of the table, the experimental results of the present scheme when the parameters of the image enhancement method and the target detection algorithm are optimized at the same time, the mAP is further improved and can reach 5.12%.

[0110] On the other hand, the image processing method proposed by the present scheme can achieve a processing speed of 300 FPS.

[0111] Referring to Figure 7 Fig. 1 is a structural schematic diagram of an image processing device provided by an embodiment of the present application. As shown in Figure 7 The device can include a processing module 701 and a fusion module 702, specifically as follows:

[0112] The processing module 701 is configured to process an initial image to obtain a semantic interest region map of the initial image, N enhanced processed images, and N fusion weight coefficient maps, where N is an integer greater than or equal to 2.

[0113] The fusion module 702 is configured to obtain a fused enhanced image according to the semantic interest region map, the N fusion weight coefficient maps, and the N enhanced processed images.

[0114] In a possible implementation, the fusion module 702 is configured to: obtain N semantic fusion weight coefficient maps according to the semantic interest region map and the N fusion weight coefficient maps, the N semantic fusion weight coefficient maps corresponding to the N fusion weight coefficient maps; and perform fusion processing on the N enhanced processed images according to the N semantic fusion weight coefficient maps to obtain the fused enhanced image.

[0115] In a possible implementation, the processing module 701 is further configured to perform smoothing processing on the N semantic fusion weight coefficient maps respectively to obtain N smoothed semantic fusion weight coefficient maps.

[0116] The fusion module 702 is further configured to perform fusion processing on the N enhanced processed images according to the N smoothed semantic fusion weight coefficient maps to obtain the fused enhanced image.

[0117] In another possible implementation, the fusion module 702 is configured to obtain a fused initial enhanced image according to the N fusion weight coefficient maps and the N enhanced processed images, and obtain the fused enhanced image according to the fused initial enhanced image and the semantic interest region map.

[0118] In a possible implementation, the processing module 701 is configured to perform N times of enhancement processing on an initial image in parallel to obtain N enhanced processed images and N fusion weight coefficient maps, the N enhanced processed images corresponding to the N times of enhancement processing, and the N fusion weight coefficient maps corresponding to the N times of enhancement processing.

[0119] In the embodiments of the present application, a semantic interest region map of an initial image, N enhanced processed images, and N fusion weight coefficient maps are obtained by processing the initial image. Then, the semantic interest region map, the N enhanced processed images, and the N fusion weight coefficient maps are fused to obtain a fused enhanced image. By combining ROI semantic information and enhanced processed image information, the ROI region prior information is obtained in the image enhancement process, which helps to improve the visual task accuracy.

[0120] In the embodiments, the image processing apparatus is in the form of a module. The "module" can refer to an application-specific integrated circuit (ASIC), a processor and a memory executing one or more software or firmware programs, an integrated logic circuit, and / or other devices that can provide the above functions.

[0121] In addition, the processing module 701 and the fusion module 702 can be implemented by the processor 802 of the image processing apparatus shown in the figure. Figure 8

[0122] Figure 8 FIG. 7 is a hardware structure schematic diagram of an image processing apparatus provided by the embodiments of the present application. Figure 8 ​The illustrated image processing apparatus 800 (which can be a computer device in particular) comprises a memory 801, a processor 802, a communication interface 803 and a bus 804. The memory 801, the processor 802 and the communication interface 803 are communicatively connected to each other through the bus 804.

[0123] The memory 801 can be a read-only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM).

[0124] The memory 801 can store a program, and when the program stored in the memory 801 is executed by the processor 802, the processor 802 and the communication interface 803 are configured to perform various steps of the image processing method of the embodiments of the present application.

[0125] The processor 802 can be a general-purpose central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU) or one or more integrated circuits, configured to execute related programs to implement the functions required by the units in the image processing apparatus of the embodiments of the present application, or to execute the image processing method of the method embodiments of the present application.

[0126] The processor 802 can also be an integrated circuit chip on which one or more of the above described functions are performed. In implementation, the various steps of the image processing method of the present application can be completed by integrated logic circuits or instructions in the form of software in the processor 802. The processor 802 described above can also be a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components. The various methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as a hardware code processor to execute, or be executed by a combination of hardware and software modules in the code processor. The software module can be located in a random access memory, a flash memory, a read only memory, a programmable read only memory or an electrically erasable programmable memory, a register, etc. The storage medium in the art is mature. The storage medium is located in the memory 801, and the processor 802 reads the information in the memory 801, and combines the hardware to complete the functions required by the units included in the image processing device of the embodiments of the present application, or executes the image processing method of the method embodiments of the present application.

[0127] The communication interface 803 uses a transceiver device such as but not limited to a transceiver to realize the communication between the device 800 and other devices or communication networks. For example, data can be obtained through the communication interface 803.

[0128] The bus 804 can include a path for transmitting information between various components (for example, the memory 801, the processor 802, the communication interface 803) of the device 800.

[0129] It should be noted that although Figure 8 The device 800 shown only shows the memory, the processor, the communication interface, but in the specific implementation process, those skilled in the art should understand that the device 800 also includes other devices necessary for normal operation. At the same time, according to the specific needs, those skilled in the art should understand that the device 800 can also include hardware devices for realizing other additional functions. In addition, those skilled in the art should understand that the device 800 can also only include the devices necessary for the embodiments of the present application, and does not have to include all the devices shown in the above. Figure 8 At the same time, the device 800 can also include other devices necessary for normal operation. At the same time, according to the specific needs, those skilled in the art should understand that the device 800 can also include hardware devices for realizing other additional functions. In addition, those skilled in the art should understand that the device 800 can also only include the devices necessary for the embodiments of the present application, and does not have to include all the devices shown in the above.

[0130] The embodiment of the present application further provides a computer readable storage medium, wherein instructions are stored in the computer readable storage medium, and when the instructions are run on a computer or a processor, the computer or the processor executes one or more steps in any one of the above methods.

[0131] The embodiment of the present application further provides a computer program product containing instructions. When the computer program product is run on a computer or a processor, the computer or the processor executes one or more steps in any one of the above methods.

[0132] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described system, device and unit can refer to the specific description of the corresponding step processes in the foregoing method embodiments, which will not be described here.

[0133] It should be understood that, in the description of the present application, unless otherwise specified, " / " represents that the objects before and after the " / " are in an "or" relationship, for example, A / B can represent A or B; wherein A, B can be singular or plural. And, in the description of the present application, unless otherwise specified, "multiple" means two or more than two. "At least one of the following" or the like means any combination of the items, including any combination of single item or multiple items. For example, at least one of a, b, or c can represent: a, b, c, a-b, a-c, b-c, or a-b-c, wherein a, b, c can be single or multiple. In addition, in order to clearly describe the technical solutions of the embodiments of the present application, in the embodiments of the present application, the same items or similar items with basically the same function and role are distinguished by using "first", "second", etc. The skilled in the art can understand that "first", "second", etc. do not limit the quantity and execution order, and "first", "second", etc. also do not limit the difference. At the same time, in the embodiments of the present application, "exemplary" or "for example" is used to represent as an example, illustration or description. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Exactly, the use of "exemplary" or "for example" and the like aims to present the relevant concept in a specific way, and is convenient for understanding.

[0134] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other manners. For example, the division of the units is only a logical function division, and there can be another division manner for actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The displayed or discussed mutual couplings between or among the units, or the direct couplings or communication connections between the units, can be indirect couplings or communication connections through some interfaces, devices or units, and can be electrical, mechanical or in other forms.

[0135] The units described as separated components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purposes of the embodiments of the present application.

[0136] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed by a computer, the computer instructions generate all or part of the processes or functions according to the embodiments of the present application. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted by the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through a wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) manner. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available medium can be a read-only memory (ROM), or a random access memory (RAM), or a magnetic medium, such as a floppy disk, a hard disk, a magnetic tape, a magnetic disk, or an optical medium, such as a digital versatile disc (DVD), or a semiconductor medium, such as a solid state disk (SSD), etc.

[0137] The above description is merely a specific implementation of the embodiments of this application, but the protection scope of the embodiments of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the embodiments of this application should be covered within the protection scope of the embodiments of this application. Therefore, the protection scope of the embodiments of this application should be determined by the protection scope of the claims.

Claims

1. An image processing method, characterized in that, include: The initial image is processed to obtain a semantic interest region map of the initial image, N enhanced images, and N fusion weight coefficient maps, where N is an integer not less than 2; the N fusion weight coefficient maps are obtained by performing N independent enhancement processes on the initial image. The fused enhanced image is obtained based on the semantic interest region map, the N fusion weight coefficient maps, and the N enhanced images. The step of obtaining the fused enhanced image based on the semantic interest region map, the N fusion weight coefficient maps, and the N enhanced images includes: obtaining N semantic fusion weight coefficient maps based on the semantic interest region map and the N fusion weight coefficient maps, wherein the N semantic fusion weight coefficient maps correspond to the N fusion weight coefficient maps; and performing fusion processing on the N enhanced images based on the N semantic fusion weight coefficient maps to obtain the fused enhanced image.

2. The method according to claim 1, characterized in that, The method further includes: The N semantic fusion weight coefficient maps are smoothed to obtain N smoothed semantic fusion weight coefficient maps. The step of fusing the N enhanced images based on the N semantic fusion weight coefficient maps to obtain the fused enhanced image includes: The N enhanced images are fused based on the N smoothed semantic fusion weight coefficient maps to obtain the fused enhanced image.

3. The method according to claim 1, characterized in that, The step of obtaining the fused enhanced image based on the semantic interest region map, the N fusion weight coefficient maps, and the N enhanced images includes: The initial enhanced image after fusion is obtained based on the N fusion weight coefficient maps and the N enhanced images; The fused enhanced image is obtained based on the fused initial enhanced image and the semantic interest region map.

4. The method according to any one of claims 1 to 3, characterized in that, The process of processing the initial image to obtain N enhanced images and N fusion weight coefficient maps includes: The initial image is subjected to N enhancement processes in parallel to obtain N enhanced images and N fusion weight coefficient maps. The N enhanced images correspond to the N enhancement processes, and the N fusion weight coefficient maps correspond to the N enhancement processes.

5. An image processing apparatus, characterized in that, include: The processing module is used to process the initial image to obtain a semantic interest region map of the initial image, N enhanced images, and N fusion weight coefficient maps, where N is an integer not less than 2; the N fusion weight coefficient maps are obtained by performing N independent enhancement processes on the initial image; A fusion module is used to obtain a fused enhanced image based on the semantic interest region map, the N fusion weight coefficient maps, and the N enhanced images; wherein, the fusion module is used to: obtain N semantic fusion weight coefficient maps based on the semantic interest region map and the N fusion weight coefficient maps, wherein the N semantic fusion weight coefficient maps correspond to the N fusion weight coefficient maps; and perform fusion processing on the N enhanced images based on the N semantic fusion weight coefficient maps to obtain the fused enhanced image.

6. The apparatus according to claim 5, characterized in that, The processing module is further configured to: The N semantic fusion weight coefficient maps are smoothed to obtain N smoothed semantic fusion weight coefficient maps. The fusion module is also used for: The N enhanced images are fused based on the N smoothed semantic fusion weight coefficient maps to obtain the fused enhanced image.

7. The apparatus according to claim 5, characterized in that, The fusion module is used for: The initial enhanced image after fusion is obtained based on the N fusion weight coefficient maps and the N enhanced images; The fused enhanced image is obtained based on the fused initial enhanced image and the semantic interest region map.

8. The apparatus according to any one of claims 5 to 7, characterized in that, The processing module is used for: The initial image is subjected to N enhancement processes in parallel to obtain N enhanced images and N fusion weight coefficient maps. The N enhanced images correspond to the N enhancement processes, and the N fusion weight coefficient maps correspond to the N enhancement processes.

9. An image processing apparatus, characterized in that, It includes a processor and a memory; wherein the memory is used to store program code, and the processor is used to call the program code to perform the method as described in any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that is executed by a processor to implement the method of any one of claims 1 to 4.

11. A computer program product, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the method as described in any one of claims 1 to 4.

12. A chip system, characterized in that, The chip system is applied to an electronic device; the chip system includes one or more interface circuits and one or more processors; the interface circuits and the processors are interconnected via lines; the interface circuits are used to receive signals from the memory of the electronic device and send the signals to the processors, the signals including computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device performs the method as described in any one of claims 1 to 4.

Images