Region-of-interest enhancement processing method and device, electronic equipment and storage medium

Abstract

The invention provides a region-of-interest enhancement processing method and device, electronic equipment and a storage medium, and relates to the field of artificial intelligence such as computer vision, image processing and video encoders. The method can comprise the steps that N region-of-interest detection results of a to-be-processed image are obtained, N is a positive integer larger than one, and the N region-of-interest detection results comprise detection results obtained after region-of-interest detection is conducted on the to-be-processed image through N different detection operators; respectively generating a target enhancement mode mask pattern and a target enhancement intensity mask pattern corresponding to the to-be-processed image according to the N region-of-interest detection results; and according to the target enhancement mode mask graph and the target enhancement intensity mask graph, performing enhancement processing on the region of interest in the to-be-processed image.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a method, device, electronic device and storage medium for enhancing a region of interest in the fields of computer vision, image processing and video encoder. Background Art

[0002] At present, video has become an indispensable part of people's lives. People can conveniently watch various small videos, short videos, live broadcasts, movies or TV series through various applications (APPs). For video manufacturers, how to improve the subjective quality of videos is an urgent problem to be solved. Summary of the invention

[0003] The present disclosure provides a method, device, electronic device and storage medium for enhancing a region of interest.

[0004] A method for enhancing a region of interest, comprising:

[0005] Obtaining N types of interest region detection results of the image to be processed, where N is a positive integer greater than one, and the N types of interest region detection results include: detection results obtained by performing interest region detection on the image to be processed using N different detection operators respectively;

[0006] According to the N types of region of interest detection results, respectively generate a target enhancement mode mask map and a target enhancement intensity mask map corresponding to the image to be processed;

[0007] According to the target enhancement mode mask map and the target enhancement strength mask map, an enhancement process is performed on the region of interest in the image to be processed.

[0008] A device for enhancing a region of interest, comprising: an acquisition module, a generation module and an enhancement module;

[0009] The acquisition module is used to acquire N types of interest region detection results of the image to be processed, where N is a positive integer greater than one, and the N types of interest region detection results include: detection results obtained by performing interest region detection on the image to be processed using N different detection operators respectively;

[0010] The generating module is used to generate a target enhancement mode mask map and a target enhancement intensity mask map corresponding to the image to be processed according to the N types of region of interest detection results;

[0011] The enhancement module is used to perform enhancement processing on the region of interest in the image to be processed according to the target enhancement mode mask map and the target enhancement strength mask map.

[0012] An electronic device, comprising:

[0013] at least one processor; and

[0014] a memory communicatively connected to the at least one processor; wherein,

[0015] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the method described above.

[0016] A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the method as described above.

[0017] A computer program product comprises a computer program / instruction, wherein the computer program / instruction implements the method described above when executed by a processor.

[0018] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0020] Figure 1 This is a flowchart of a first embodiment of the method for enhancing a region of interest described in the present disclosure;

[0021] Figure 2 This is a flow chart of a second embodiment of the method for enhancing a region of interest described in the present disclosure;

[0022] Figure 3 It is a schematic diagram of the composition structure of an embodiment 300 of the device for enhancing the region of interest described in the present disclosure;

[0023] Figure 4 A schematic block diagram of an electronic device 400 that can be used to implement an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0024] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0025] In addition, it should be understood that the term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0026] Figure 1 FIG. 1 is a flow chart of the first embodiment of the method for enhancing the region of interest described in the present disclosure. Figure 1 As shown, the following specific implementation methods are included.

[0027] In step 101, N types of region of interest (ROI) detection results of the image to be processed are obtained, where N is a positive integer greater than one, and the N types of ROI area detection results include: detection results obtained by performing ROI area detection on the image to be processed using N different detection operators.

[0028] In step 102, according to the N types of ROI region detection results, a target enhancement mode mask map and a target enhancement intensity mask map corresponding to the image to be processed are generated respectively.

[0029] In step 103, the ROI region in the image to be processed is enhanced according to the target enhancement mode mask map and the target enhancement strength mask map.

[0030] By adopting the scheme described in the above method embodiment, a target enhancement mode mask map and a target enhancement strength mask map can be generated according to the N ROI area detection results of the image to be processed, and the ROI area in the image to be processed can be enhanced according to the two mask maps. The image to be processed can be an image in the video, thereby improving the quality of the user's focus area, thereby improving the subjective quality of the video, etc.

[0031] The specific value of N can be determined according to actual needs. N different detection operators can be used to perform ROI area detection on the image to be processed, thereby obtaining N ROI area detection results. For example, commonly used detection operators may include face detection operators, human body detection operators, subtitle detection operators, salient area detection operators, and contour detection operators, etc. The corresponding ROI area detection results may be: face frame, human body frame, subtitle frame, salient area mask map, and contour point (i.e., contour pixel position set). In practical applications, the number of ROI area detection results obtained may also be greater than N, and some of them, such as N, may be selected for processing according to the scheme described in the present disclosure, and the remaining ROI area detection results may be used for other processing.

[0032] According to the N types of ROI region detection results, a target enhancement mode mask map and a target enhancement intensity mask map corresponding to the image to be processed can be generated respectively.

[0033] In some embodiments of the present disclosure, an initial enhancement mode mask map can be generated according to the enhancement mode corresponding to the image to be processed and the preset image to be processed, and the initial enhancement mode mask map can be assigned and optimized according to the N ROI area detection results and the enhancement modes corresponding to the preset N ROI area detection results, to obtain a target enhancement mode mask map. In addition, an initial enhancement intensity mask map can be generated according to the image to be processed and the enhancement intensity corresponding to the preset image to be processed, and the initial enhancement intensity mask map can be assigned and optimized according to the enhancement intensity corresponding to the N ROI area detection results and the preset N ROI area detection results, to obtain a target enhancement intensity mask map. The preset setting can be manually pre-configured and can be adjusted at any time according to actual needs.

[0034] In some embodiments of the present disclosure, the size of the initial enhancement mode mask map is the same as that of the image to be processed, and the value of each pixel in the initial enhancement mode mask map is a first initial value, which is a numerical representation of the enhancement mode corresponding to the image to be processed. Similarly, the size of the initial enhancement strength mask map is the same as that of the image to be processed, and the value of each pixel in the initial enhancement strength mask map is a second initial value, which is a numerical representation of the enhancement strength corresponding to the image to be processed.

[0035] That is to say, in the solution disclosed in the present invention, the image to be processed (the whole frame image) will correspond to its own enhancement mode and enhancement strength, and at the same time, various ROI area detection results will also correspond to their own enhancement modes and enhancement strengths.

[0036] Accordingly, two mask images, namely, the initial enhancement mode mask image and the initial enhancement strength mask image, can be first constructed according to the whole frame mode, the size of which is the same as the image to be processed, the value of each pixel in the initial enhancement mode mask image is equal to the numerical representation of the enhancement mode corresponding to the image to be processed, and the value of each pixel in the initial enhancement strength mask image is equal to the numerical representation of the enhancement strength corresponding to the image to be processed. Afterwards, the initial enhancement mode mask image and the initial enhancement strength mask image can be assigned and optimized according to the N types of ROI area detection results and the enhancement modes and enhancement strengths corresponding to the N types of ROI area detection results, respectively, so as to obtain the required target enhancement mode mask image and target enhancement strength mask image.

[0037] In traditional methods, usually only a single ROI region is processed, such as only a face, or the face, body and subtitles are combined into one ROI region for processing, and then an enhancement strategy is configured for the ROI region, resulting in poor enhancement effect. However, the scheme described in the present disclosure can realize multi-ROI region enhancement based on mask images, and set corresponding enhancement modes and enhancement strengths for the image to be processed and different types of ROI region detection results, thereby improving the enhancement effect and thus improving the image quality.

[0038] In addition, whether it is the initial enhancement mode mask map or the target enhancement mode mask map, the value of each pixel therein can be represented by an 8-bit unsigned 8-bit integer (uint8_t), and its value can correspond to an enhancement mode. The enhancement mode can include only a specific enhancement operator, or a specific enhancement operator and related control information of the enhancement operator, such as the switch of the sub-strategy in the enhancement operator, etc. For example, the enhancement operator can be represented by part of the bits in the 8bit, and the related control information can be represented by the remaining part of the bits, which is very flexible and convenient. The numerical representations corresponding to different enhancement modes can be determined according to actual needs. The enhancement operator can refer to operators such as blur masking and sharpening (USM, UnsharpMask). In addition, whether it is the initial enhancement strength mask map or the target enhancement strength mask map, each pixel therein can be a value of an 8-bit integer (int8_t) or a 16-bit integer (int16_t), corresponding to the integer enhancement strength.

[0039] For the initial enhancement mode mask map and the initial enhancement strength mask map, the target enhancement mode mask map and the target enhancement strength mask map can be obtained by respectively performing value optimization on them.

[0040] In some embodiments of the present disclosure, a method for optimizing the assignment of the initial enhancement mode mask map may include: traversing N types of ROI area detection results in order from low to high priority according to a preset priority, and performing the following processing for each traversed ROI area detection result: taking the currently traversed ROI area detection result as the first detection result to be processed, determining the pixel points that match the first detection result to be processed from the initial enhancement mode mask map, and setting the values ​​of the matching pixel points to the numerical representation of the enhancement mode corresponding to the first detection result to be processed; similarly, a method for optimizing the assignment of the initial enhancement strength mask map may include: traversing N types of ROI area detection results in order from low to high priority, and performing the following processing for each traversed ROI area detection result: taking the currently traversed ROI area detection result as the second detection result to be processed, determining the pixel points that match the second detection result to be processed from the initial enhancement strength mask map, and setting the values ​​of the matching pixel points to the numerical representation of the enhancement strength corresponding to the second detection result to be processed.

[0041] There is no restriction on the priority order of N ROI region detection results. For example, assuming that the N ROI region detection results include: face frame, body frame, subtitle frame, salient region mask map and contour point, then the order from low to high priority can be: salient region mask map, contour point, subtitle frame, body frame and face frame.

[0042] Taking the assignment optimization of the initial enhancement mode mask map as an example, the ROI area detection result with the lowest priority can be first used as the first detection result to be processed, and the pixel points matching the first detection result to be processed can be determined from the initial enhancement mode mask map, and then the values ​​of the matching pixel points can be set to the numerical representation of the enhancement mode corresponding to the first detection result to be processed (at this time, the ROI area detection result with the lowest priority). After that, the ROI area detection result with the second lowest priority can be used as the first detection result to be processed, and the pixel points matching the first detection result to be processed can be determined from the initial enhancement mode mask map, and then the values ​​of the matching pixel points can be set to the numerical representation of the enhancement mode corresponding to the first detection result to be processed (at this time, the ROI area detection result with the second lowest priority), and so on, until the above processing is completed for the ROI area detection result with the highest priority to obtain the target enhancement mode mask map.

[0043] Since the processing is performed in order from low to high priority, the high-priority assignment result will overwrite the low-priority assignment result. In this way, even if a pixel point matches multiple ROI area detection results, it will eventually be assigned according to the ROI area detection result with the highest priority, thereby ensuring the accuracy of the processing results.

[0044] In addition, different types of ROI region detection results can be expressed in different forms. For example, face frames, human frames, and subtitle frames are all detection results in the form of rectangular frames. Contour points are a set of contour pixel positions. The salient region mask is a mask that is marked with significant (indicated by 1) and non-significant (indicated by 0) in a frame. In the traditional way, when merging different types of ROI region detection results into one ROI region, it is usually necessary to perform multiple processing, such as first processing the face frame, then processing the subtitle frame, contour points, and salient region mask, and then processing the entire image, and finally merging the various processing results into one image. The process is very complicated, and the processing logic and computational complexity are very high. Moreover, a certain pixel may have multiple attributes, such as being both a pixel on the face and a contour point pixel, which can be very troublesome to merge. In the solution disclosed in the present invention, it is only necessary to assign values ​​according to the priority order of different types of ROI area detection results to obtain the required target enhancement mode mask map. The whole process is simple and convenient to implement, thereby effectively simplifying the processing logic, reducing the calculation complexity, and improving the processing efficiency.

[0045] Among them, when determining the pixel points matching the first detection result to be processed from the initial enhancement mode mask image, corresponding methods can be used to determine the matching pixel points for different types of ROI area detection results, and the methods mainly include the following method 1, method 2 and method 3.

[0046] 1) Method 1

[0047] In response to determining that the first detection result to be processed is a detection result in the form of a rectangular frame, the pixel points in the initial enhancement mode mask image that are located in the rectangular area corresponding to the rectangular frame may be determined as matching pixel points.

[0048] The detection results in the form of rectangular frames may include: face frames, body frames, and subtitle frames.

[0049] Taking the face frame as an example, assuming that the pixel with coordinate position (12, 15) in the initial enhancement mode mask image is located in the rectangular area corresponding to the face frame, then the pixel can be determined as a matching pixel, and the value of the matching pixel can be set as the numerical representation of the enhancement mode corresponding to the face frame.

[0050] 2) Method 2

[0051] In response to determining that the first detection result to be processed is a salient area mask map, the corresponding pixel points in the initial enhancement mode mask map of the pixel points with a value of 1 in the salient area mask map can be determined as matching pixel points, the size of the salient area mask map is the same as the image to be processed, and the value of each pixel point in the salient area mask map is 1 or 0, and the corresponding pixel points are pixel points with the same coordinate position.

[0052] For example, if the value of the pixel at the coordinate position (2, 4) in the salient region mask is 1, then the pixel at the same coordinate position in the initial enhancement mode mask can be determined as a matching pixel, and then the value of the matching pixel can be set as the numerical representation of the enhancement mode corresponding to the salient region mask.

[0053] 3) Method 3

[0054] In response to determining that the first detection result to be processed is a contour point, the pixel points belonging to the contour point in the initial enhancement mode mask image can be determined as matching pixel points, or the pixel points belonging to the contour point in the initial enhancement mode mask image and the pixel points adjacent to the pixel points belonging to the contour point can be determined as matching pixel points.

[0055] For example, the pixel point with coordinate position (20, 20) in the initial enhancement mode mask image belongs to the contour point, so the pixel point can be determined as a matching pixel point, or the pixel point and the pixel points adjacent to the pixel point (up, down, left, and right) can be determined as matching pixels, and the value of the matching pixel point can be set to the numerical representation of the enhancement mode corresponding to the contour point.

[0056] It can be seen that in the above processing method, for different types of ROI area detection results, corresponding methods can be used to determine matching pixels from the initial enhancement mode mask image, thereby improving the accuracy of the determination results, and further improving the accuracy of the subsequent target enhancement mode mask image.

[0057] Similarly, when determining the pixel points that match the second detection result to be processed from the initial enhanced intensity mask image, corresponding methods can be used to determine the matching pixel points for different types of ROI area detection results, and the methods mainly include the following method 1, method 2 and method 3.

[0058] 1) Method 1

[0059] In response to determining that the second detection result to be processed is a detection result in the form of a rectangular frame, the pixel points in the initial enhanced intensity mask image that are located in the rectangular area corresponding to the rectangular frame may be determined as matching pixel points.

[0060] The detection results in the form of rectangular frames may include: face frames, body frames, and subtitle frames.

[0061] 2) Method 2

[0062] In response to determining that the second detection result to be processed is a salient region mask map, corresponding pixels of pixels having a value of 1 in the salient region mask map in the initial enhanced strength mask map may be determined as matching pixels.

[0063] 3) Method 3

[0064] In response to determining that the second detection result to be processed is a contour point, the pixel points belonging to the contour point in the initial enhanced intensity mask image can be determined as matching pixel points, or the pixel points belonging to the contour point in the initial enhanced intensity mask image and the pixel points adjacent to the pixel points belonging to the contour point can be determined as matching pixel points.

[0065] Regardless of method one, method two or method three, after determining the pixel points that match the second detection result to be processed from the initial enhanced intensity mask image, the value of the matching pixel points can be set to the numerical representation of the enhanced intensity corresponding to the second detection result to be processed.

[0066] It can be seen that in the above processing method, for different types of ROI area detection results, corresponding methods can be used to determine matching pixels from the initial enhanced intensity mask map, thereby improving the accuracy of the determination results, and further improving the accuracy of the subsequent target enhanced intensity mask map.

[0067] After respectively obtaining the target enhancement mode mask map and the target enhancement strength mask map, the ROI region in the image to be processed can be enhanced according to the two mask maps.

[0068] In some embodiments of the present disclosure, the following processing can be performed for each pixel point in the image to be processed: obtain a first value of the corresponding pixel point of the pixel point in the target enhancement mode mask image, and obtain a second value of the corresponding pixel point of the pixel point in the target enhancement strength mask image, the corresponding pixel points are pixel points with the same coordinate position, and the pixel point is enhanced according to the enhancement mode corresponding to the first value and the enhancement strength corresponding to the second value.

[0069] That is, a pixel-level processing method can be adopted to traverse each pixel point in the image to be processed, and when traversing each pixel point, the value of the corresponding pixel point in the target enhancement mode mask map and the target enhancement strength mask map can be obtained respectively, so that the enhancement mode and enhancement strength of the pixel point can be obtained respectively, and then the pixel point can be enhanced according to the acquired enhancement mode and enhancement strength.

[0070] By adopting the above processing method, the enhancement processing of the image to be processed can be completed according to the target enhancement mode mask map and the target enhancement strength mask map. The enhancement processing process is decoupled from the ROI area detection result, thereby simplifying the processing logic and improving the processing efficiency. Moreover, not only the enhancement of the ROI area in the image to be processed can be achieved, but also the enhancement mode and enhancement strength are set for the image to be processed, so the enhancement of the non-ROI area can be achieved, thereby further improving the enhancement effect.

[0071] Combined with the above introduction, Figure 2 FIG. 2 is a flow chart of the second embodiment of the method for enhancing the region of interest described in the present disclosure. Figure 2 As shown, the following specific implementation methods are included.

[0072] In step 201, N ROI region detection results of the image to be processed are obtained, where N is a positive integer greater than one, and the N ROI region detection results include: detection results obtained by performing ROI region detection on the image to be processed using N different detection operators.

[0073] In step 202, an initial enhancement mode mask image is generated according to the image to be processed and the enhancement mode corresponding to the image to be processed that is preset. The size of the initial enhancement mode mask image is the same as that of the image to be processed, and the value of each pixel point therein is a first initial value, which is a numerical representation of the enhancement mode corresponding to the image to be processed.

[0074] In step 203, N types of ROI region detection results are traversed in order from low to high according to the preset priorities, and the first traversed ROI region detection result is used as the first detection result to be processed.

[0075] In step 204, pixel points that match the first detection result to be processed are determined from the initial enhancement mode mask image, and the values ​​of the matching pixel points are set as the numerical representation of the enhancement mode corresponding to the first detection result to be processed.

[0076] In step 205 , it is determined whether there is an untraversed ROI region detection result, if so, step 206 is executed, otherwise, step 207 is executed.

[0077] In step 206, the next ROI region detection result is used as the first detection result to be processed, and then step 204 is repeated.

[0078] In step 207 , the latest assigned initial enhancement mode mask image is determined as the target enhancement mode mask image, and then step 208 is executed.

[0079] In step 208, an initial enhancement intensity mask image is generated according to the image to be processed and the enhancement intensity corresponding to the image to be processed which is preset. The size of the initial enhancement intensity mask image is the same as that of the image to be processed, and the value of each pixel point therein is a second initial value, which is a numerical representation of the enhancement intensity corresponding to the image to be processed.

[0080] In step 209, N types of ROI region detection results are traversed in order from low to high according to the preset priorities, and the first traversed ROI region detection result is used as the second detection result to be processed.

[0081] In step 210, pixel points matching the second detection result to be processed are determined from the initial enhanced intensity mask image, and the values ​​of the matching pixel points are set as the numerical representation of the enhanced intensity corresponding to the second detection result to be processed.

[0082] In step 211 , it is determined whether there is an untraversed ROI region detection result, if so, step 212 is executed, otherwise, step 213 is executed.

[0083] In step 212, the next ROI region detection result is used as the second detection result to be processed, and then step 210 is repeated.

[0084] In step 213 , the latest assigned initial enhanced intensity mask map is determined as the target enhanced intensity mask map, and then step 214 is executed.

[0085] In step 214, for each pixel in the image to be processed, the first value of the corresponding pixel in the target enhancement mode mask image is obtained, and the second value of the corresponding pixel in the target enhancement strength mask image is obtained. The pixel is enhanced according to the enhancement mode corresponding to the first value and the enhancement strength corresponding to the second value, and then the process ends.

[0086] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present disclosure is not limited by the described order of actions, because according to the present disclosure, certain steps can be performed in other orders or simultaneously, for example, steps 202-207 can be performed simultaneously with steps 208-213. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present disclosure. In addition, for parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0087] The above is an introduction to the method embodiment. The following is a further explanation of the scheme disclosed in the present invention through an apparatus embodiment.

[0088] Figure 3 FIG. 3 is a schematic diagram of the structure of the ROI region enhancement processing device embodiment 300 described in the present disclosure. Figure 3 As shown, it includes: an acquisition module 301, a generation module 302 and an enhancement module 303.

[0089] The acquisition module 301 is used to acquire N ROI region detection results of the image to be processed, where N is a positive integer greater than one, and the N ROI region detection results include: detection results obtained by performing ROI region detection on the image to be processed using N different detection operators.

[0090] The generating module 302 is used to generate a target enhancement mode mask map and a target enhancement intensity mask map corresponding to the image to be processed according to the N types of ROI region detection results.

[0091] The enhancement module 303 is used to perform enhancement processing on the ROI region in the image to be processed according to the target enhancement mode mask map and the target enhancement strength mask map.

[0092] By adopting the scheme described in the above-mentioned device embodiment, a target enhancement mode mask map and a target enhancement strength mask map can be generated according to the N ROI area detection results of the image to be processed, and the ROI area in the image to be processed can be enhanced according to the two mask maps. The image to be processed can be an image in the video, thereby improving the quality of the area of ​​user attention, and further improving the subjective quality of the video.

[0093] The specific value of N can be determined according to actual needs. N different detection operators can be used to perform ROI region detection on the image to be processed, thereby obtaining N ROI region detection results. For example, commonly used detection operators may include face detection operators, human body detection operators, subtitle detection operators, salient region detection operators, and contour detection operators, and the corresponding ROI region detection results may be: face frame, human body frame, subtitle frame, salient region mask map, and contour points.

[0094] According to the N types of ROI region detection results, the generation module 302 can respectively generate a target enhancement mode mask map and a target enhancement intensity mask map corresponding to the image to be processed.

[0095] In some embodiments of the present disclosure, the generation module 302 may include: a first generation unit 3021 and a second generation unit 3022 .

[0096] The first generating unit 3021 is used to generate an initial enhancement mode mask map according to the image to be processed and the enhancement mode corresponding to the pre-set image to be processed, and to optimize the initial enhancement mode mask map according to the N types of ROI area detection results and the enhancement modes corresponding to the pre-set N types of ROI area detection results, so as to obtain a target enhancement mode mask map.

[0097] The second generation unit 3022 is used to generate an initial enhancement intensity mask map according to the image to be processed and the enhancement intensity corresponding to the preset image to be processed, and to optimize the initial enhancement intensity mask map according to the enhancement intensity corresponding to the N types of ROI area detection results and the preset N types of ROI area detection results, so as to obtain a target enhancement intensity mask map.

[0098] In addition, in some embodiments of the present disclosure, the size of the initial enhancement mode mask map is the same as that of the image to be processed, and the value of each pixel in the initial enhancement mode mask map is a first initial value, which is a numerical representation of the enhancement mode corresponding to the image to be processed. Similarly, the size of the initial enhancement strength mask map is the same as that of the image to be processed, and the value of each pixel in the initial enhancement strength mask map is a second initial value, which is a numerical representation of the enhancement strength corresponding to the image to be processed.

[0099] In some embodiments of the present disclosure, the first generation unit 3021 may optimize the assignment of the initial enhancement mode mask map by the first generation unit 3021, which may include: traversing N types of ROI area detection results in order from low to high according to a preset priority, and performing the following processing for each traversed ROI area detection result: taking the currently traversed ROI area detection result as the first detection result to be processed, determining the pixel points that match the first detection result to be processed from the initial enhancement mode mask map, and setting the values ​​of the matching pixel points to the numerical representation of the enhancement mode corresponding to the first detection result to be processed; similarly, the second generation unit 3022 may optimize the assignment of the initial enhancement strength mask map by the second generation unit 3022, which may include: traversing N types of ROI area detection results in order from low to high according to a priority, and performing the following processing for each traversed ROI area detection result: taking the currently traversed ROI area detection result as the second detection result to be processed, determining the pixel points that match the second detection result to be processed from the initial enhancement strength mask map, and setting the values ​​of the matching pixel points to the numerical representation of the enhancement strength corresponding to the second detection result to be processed.

[0100] In addition, in some embodiments of the present disclosure, in response to determining that the first detection result to be processed is a detection result in the form of a rectangular box, the first generation unit 3021 may determine the pixel points in the initial enhancement mode mask image that are located in the rectangular area corresponding to the rectangular box as matching pixel points. Similarly, in response to determining that the second detection result to be processed is a detection result in the form of a rectangular box, the second generation unit 3022 may determine the pixel points in the initial enhancement intensity mask image that are located in the rectangular area corresponding to the rectangular box as matching pixel points.

[0101] In some embodiments of the present disclosure, in response to determining that the first detection result to be processed is a salient area mask map, the first generation unit 3021 may determine the corresponding pixel points in the initial enhancement mode mask map of the pixel points with a value of 1 in the salient area mask map as matching pixel points, the size of the salient area mask map is the same as the image to be processed, and the value of each pixel point in the salient area mask map is 1 or 0, respectively, and the corresponding pixel points are pixel points with the same coordinate position. Similarly, in response to determining that the second detection result to be processed is a salient area mask map, the second generation unit 3022 may determine the corresponding pixel points in the initial enhancement strength mask map of the pixel points with a value of 1 in the salient area mask map as matching pixel points.

[0102] In some embodiments of the present disclosure, in response to determining that the first detection result to be processed is a contour point, the first generation unit 3021 may determine the pixel points belonging to the contour point in the initial enhancement mode mask image as matching pixel points, or determine the pixel points belonging to the contour point in the initial enhancement mode mask image and the pixel points adjacent to the pixel points belonging to the contour point as matching pixel points. Similarly, in response to determining that the second detection result to be processed is a contour point, the second generation unit 3022 may determine the pixel points belonging to the contour point in the initial enhancement strength mask image as matching pixel points, or determine the pixel points belonging to the contour point in the initial enhancement strength mask image and the pixel points adjacent to the pixel points belonging to the contour point as matching pixel points.

[0103] After respectively acquiring the target enhancement mode mask map and the target enhancement strength mask map, the enhancement module 303 may perform enhancement processing on the ROI region in the image to be processed according to the two mask maps.

[0104] In some embodiments of the present disclosure, the enhancement module 303 may perform the following processing on each pixel point in the image to be processed: obtain a first value of the corresponding pixel point of the pixel point in the target enhancement mode mask image, and obtain a second value of the corresponding pixel point of the pixel point in the target enhancement strength mask image, where the corresponding pixel points are pixel points with the same coordinate position, and the pixel point is enhanced according to the enhancement mode corresponding to the first value and the enhancement strength corresponding to the second value.

[0105] Figure 3 The specific working process of the illustrated device embodiment can refer to the relevant description in the aforementioned method embodiment and will not be described in detail.

[0106] The solution disclosed in the present disclosure can be applied to the field of artificial intelligence, especially computer vision, image processing, video encoders and other fields. Artificial intelligence is a discipline that studies how to use computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It has both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing. Artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, machine learning / deep learning, big data processing technology, knowledge graph technology and other major directions.

[0107] The images and ROI area detection results in the embodiments of the present disclosure are not targeted at a specific user and do not reflect the personal information of a specific user. In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good customs.

[0108] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0109] Figure 4 A schematic block diagram of an electronic device 400 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0110] like Figure 4 As shown, the device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 408 to a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the device 400 can also be stored. The computing unit 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0111] A number of components in the device 400 are connected to the I / O interface 405, including: an input unit 406, such as a keyboard, a mouse, etc.; an output unit 407, such as various types of displays, speakers, etc.; a storage unit 408, such as a disk, an optical disk, etc.; and a communication unit 409, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 409 allows the device 400 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0112] The computing unit 401 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI, Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSP, Digital Signal Processing), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 401 performs the various methods and processes described above, such as the methods described in the present disclosure. For example, in some embodiments, the methods described in the present disclosure may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 400 via the ROM 402 and / or the communication unit 409. When the computer program is loaded into the RAM 403 and executed by the computing unit 401, one or more steps of the methods described in the present disclosure may be performed. Alternatively, in other embodiments, the computing unit 401 may be configured to execute the method described in the present disclosure in any other appropriate manner (for example, by means of firmware).

[0113] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard parts (ASSPs), system on chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0114] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0115] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory (EPROM, Electronically Programmable Read-Only Memory), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM, Compact Disc Read-Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0116] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0117] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by digital data communication (e.g., a communication network) in any form or medium. Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0118] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0119] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0120] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for enhancing a region of interest, comprising: Obtaining N types of interest region detection results of the image to be processed, where N is a positive integer greater than one, and the N types of interest region detection results include: detection results obtained by performing interest region detection on the image to be processed using N different detection operators respectively; According to the N types of region of interest detection results, respectively generate a target enhancement mode mask map and a target enhancement intensity mask map corresponding to the image to be processed; According to the target enhancement mode mask map and the target enhancement strength mask map, an enhancement process is performed on the region of interest in the image to be processed.

2. The method according to claim 1, wherein: The step of respectively generating a target enhancement mode mask map and a target enhancement intensity mask map corresponding to the image to be processed comprises: According to the image to be processed and the preset enhancement mode corresponding to the image to be processed, an initial enhancement mode mask map is generated, and according to the N types of region of interest detection results and the preset enhancement modes respectively corresponding to the N types of region of interest detection results, the initial enhancement mode mask map is assigned and optimized to obtain the target enhancement mode mask map; An initial enhancement intensity mask map is generated according to the image to be processed and the enhancement intensity corresponding to the image to be processed set in advance. According to the N types of region of interest detection results and the enhancement intensity corresponding to the N types of region of interest detection results set in advance, the initial enhancement intensity mask map is assigned and optimized to obtain the target enhancement intensity mask map.

3. The method according to claim 2, wherein: The size of the initial enhancement mode mask image is the same as that of the image to be processed, and the value of each pixel in the initial enhancement mode mask image is a first initial value, which is a numerical representation of the enhancement mode corresponding to the image to be processed; The size of the initial enhanced intensity mask image is the same as that of the image to be processed, and the value of each pixel in the initial enhanced intensity mask image is a second initial value, which is a numerical representation of the enhanced intensity corresponding to the image to be processed.

4. The method according to claim 3, wherein: The optimizing the initial enhancement mode mask map by assigning values ​​includes: The N types of interest region detection results are sequentially traversed in an ascending order of a preset priority, and the following processing is performed on each traversed interest region detection result: the currently traversed interest region detection result is used as a first detection result to be processed, a pixel point matching the first detection result to be processed is determined from the initial enhancement mode mask map, and the value of the matching pixel point is set as the numerical representation of the enhancement mode corresponding to the first detection result to be processed; The optimizing the initial enhanced strength mask map by assigning a value comprises: The N types of region of interest detection results are traversed in order from low to high priority, and the following processing is performed on each traversed region of interest detection result: the currently traversed region of interest detection result is used as the second detection result to be processed, and the pixel points that match the second detection result to be processed are determined from the initial enhancement intensity mask map, and the values ​​of the matching pixel points are set to the numerical representation of the enhancement intensity corresponding to the second detection result to be processed.

5. The method according to claim 4, wherein: In response to determining that the first detection result to be processed is a detection result in the form of a rectangular frame, determining the pixel points that match the first detection result to be processed from the initial enhancement mode mask image includes: determining the pixel points in the initial enhancement mode mask image that are located in the rectangular area corresponding to the rectangular frame as the matching pixel points; In response to determining that the second detection result to be processed is a detection result in the form of the rectangular box, determining the pixel points that match the second detection result to be processed from the initial enhanced intensity mask image includes: determining the pixel points in the initial enhanced intensity mask image that are located in the rectangular area corresponding to the rectangular box as the matching pixel points.

6. The method according to claim 4, wherein: In response to determining that the first detection result to be processed is a salient area mask map, determining the pixel points that match the first detection result to be processed from the initial enhancement mode mask map includes: determining the corresponding pixel points in the initial enhancement mode mask map of the pixel points whose values ​​in the salient area mask map are 1 as the matched pixel points, the size of the salient area mask map is the same as that of the image to be processed, and the values ​​of each pixel point in the salient area mask map are 1 or 0 respectively, and the corresponding pixel points are pixel points with the same coordinate position; In response to determining that the second detection result to be processed is the salient area mask map, determining the pixel points that match the second detection result to be processed from the initial enhanced strength mask map includes: determining the corresponding pixel points in the initial enhanced strength mask map of the pixel points with a value of 1 in the salient area mask map as the matching pixel points.

7. The method according to claim 4, wherein: In response to determining that the first detection result to be processed is a contour point, determining the pixel points that match the first detection result to be processed from the initial enhancement mode mask image includes: determining the pixel points belonging to the contour point in the initial enhancement mode mask image as the matched pixel points, or determining the pixel points belonging to the contour point in the initial enhancement mode mask image and the pixel points adjacent to the pixel points belonging to the contour point as the matched pixel points; In response to determining that the second detection result to be processed is the contour point, determining the pixel points that match the second detection result to be processed from the initial enhanced intensity mask image includes: determining the pixel points belonging to the contour point in the initial enhanced intensity mask image as the matching pixel points, or determining the pixel points belonging to the contour point in the initial enhanced intensity mask image and the pixel points adjacent to the pixel points belonging to the contour point as the matching pixel points.

8. The method according to any one of claims 4 to 7, wherein: The enhancing process of the region of interest in the image to be processed comprises: For each pixel in the image to be processed, the following processing is performed respectively: Obtaining a first value of a corresponding pixel point of the pixel point in the target enhancement mode mask image, and obtaining a second value of a corresponding pixel point of the pixel point in the target enhancement strength mask image, wherein the corresponding pixel points are pixel points with the same coordinate position; The pixel point is enhanced according to the enhancement mode corresponding to the first value and the enhancement strength corresponding to the second value.

9. A device for enhancing a region of interest, comprising: Acquisition modules, generation modules, and enhancement modules; The acquisition module is used to acquire N types of interest region detection results of the image to be processed, where N is a positive integer greater than one, and the N types of interest region detection results include: detection results obtained by performing interest region detection on the image to be processed using N different detection operators respectively; The generating module is used to generate a target enhancement mode mask map and a target enhancement intensity mask map corresponding to the image to be processed according to the N types of region of interest detection results; The enhancement module is used to perform enhancement processing on the region of interest in the image to be processed according to the target enhancement mode mask map and the target enhancement strength mask map.

10. The device according to claim 9, wherein: The generation module includes: a first generation unit and a second generation unit; The first generating unit is used to generate an initial enhancement mode mask map according to the image to be processed and a preset enhancement mode corresponding to the image to be processed, and to optimize the initial enhancement mode mask map according to the N types of region of interest detection results and the preset enhancement modes respectively corresponding to the N types of region of interest detection results, so as to obtain the target enhancement mode mask map; The second generating unit is used to generate an initial enhancement intensity mask map according to the image to be processed and the enhancement intensity corresponding to the image to be processed set in advance, and to optimize the initial enhancement intensity mask map according to the N types of region of interest detection results and the enhancement intensity corresponding to the N types of region of interest detection results set in advance, so as to obtain the target enhancement intensity mask map.

11. The device according to claim 10, wherein: The size of the initial enhancement mode mask image is the same as that of the image to be processed, and the value of each pixel in the initial enhancement mode mask image is a first initial value, which is a numerical representation of the enhancement mode corresponding to the image to be processed; The size of the initial enhanced intensity mask image is the same as that of the image to be processed, and the value of each pixel in the initial enhanced intensity mask image is a second initial value, which is a numerical representation of the enhanced intensity corresponding to the image to be processed.

12. The device according to claim 11, wherein The first generating unit sequentially traverses the N types of interest region detection results in an ascending order of preset priorities, and performs the following processing on each traversed interest region detection result: taking the currently traversed interest region detection result as the first detection result to be processed, determining a pixel point matching the first detection result to be processed from the initial enhancement mode mask map, and setting the value of the matching pixel point as the numerical representation of the enhancement mode corresponding to the first detection result to be processed; The second generation unit traverses the N types of region of interest detection results in order from low to high priority, and performs the following processing on each traversed region of interest detection result: taking the currently traversed region of interest detection result as the second detection result to be processed, determining the pixel points that match the second detection result to be processed from the initial enhancement intensity mask map, and setting the value of the matching pixel points to the numerical representation of the enhancement intensity corresponding to the second detection result to be processed.

13. The device according to claim 12, wherein: In response to determining that the first to-be-processed detection result is a detection result in the form of a rectangular frame, the first generating unit determines the pixel points in the initial enhancement mode mask image that are located in the rectangular area corresponding to the rectangular frame as the matching pixel points; In response to determining that the second detection result to be processed is a detection result in the form of the rectangular frame, the second generating unit determines the pixel points in the initial enhanced intensity mask image that are located in the rectangular area corresponding to the rectangular frame as the matching pixel points.

14. The device according to claim 12, wherein: In response to determining that the first detection result to be processed is a salient region mask map, the first generating unit determines corresponding pixel points in the initial enhancement mode mask map of pixel points having a value of 1 in the salient region mask map as the matched pixel points, the size of the salient region mask map is the same as that of the image to be processed, and the value of each pixel point in the salient region mask map is 1 or 0 respectively, and the corresponding pixel points are pixel points with the same coordinate position; In response to determining that the second detection result to be processed is the salient area mask map, the second generating unit determines corresponding pixel points of pixel points having a value of 1 in the salient area mask map in the initial enhanced strength mask map as the matched pixel points.

15. The device according to claim 12, wherein: In response to determining that the first detection result to be processed is a contour point, the first generating unit determines the pixel points belonging to the contour point in the initial enhancement mode mask image as the matched pixel points, or determines the pixel points belonging to the contour point in the initial enhancement mode mask image and the pixel points adjacent to the pixel points belonging to the contour point as the matched pixel points; In response to determining that the second detection result to be processed is the contour point, the second generation unit determines the pixel points belonging to the contour point in the initial enhanced intensity mask image as the matching pixel points, or determines the pixel points belonging to the contour point in the initial enhanced intensity mask image and the pixel points adjacent to the pixel points belonging to the contour point as the matching pixel points.

16. The device according to any one of claims 12 to 15, wherein: The enhancement module performs the following processing on each pixel point in the image to be processed: obtains a first value of a corresponding pixel point of the pixel point in the target enhancement mode mask map, and obtains a second value of a corresponding pixel point of the pixel point in the target enhancement strength mask map, the corresponding pixel points are pixel points with the same coordinate position, and the pixel point is enhanced according to the enhancement mode corresponding to the first value and the enhancement strength corresponding to the second value.

17. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to make a computer execute the method according to any one of claims 1 to 8.

19. A computer program product, comprising a computer program / instruction, wherein when the computer program / instruction is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images