Method, device, electronic device and storage medium for region of interest enhancement processing

Abstract

This disclosure provides a method, apparatus, electronic device, and storage medium for region-of-interest (ROI) enhancement processing, relating to artificial intelligence fields such as computer vision, image processing, and video encoders. The method may include: acquiring N types of ROI detection results for an image to be processed, where N is a positive integer greater than one; the N ROI detection results include: detection results obtained by performing ROI detection on the image to be processed using N different detection operators; generating a target enhancement mode mask and a target enhancement intensity mask for the image to be processed based on the N ROI detection results; and performing enhancement processing on the ROI in the image to be processed based on the target enhancement mode mask and the target enhancement intensity mask.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to methods, apparatuses, electronic devices and storage media for region-of-interest enhancement processing in the fields of computer vision, image processing and video encoders. Background Technology

[0002] Currently, video has become an indispensable part of people's lives. People can easily watch various short videos, live streams, movies, and TV series through various applications (apps). For video producers, improving the subjective quality of videos is an urgent problem to be solved. Summary of the Invention

[0003] This disclosure provides a method, apparatus, electronic device, and storage medium for region-of-interest enhancement processing.

[0004] A region of interest enhancement processing method includes:

[0005] Obtain N types of region of interest (ROI) detection results for the image to be processed, where N is a positive integer greater than one. The N types of ROI detection results include: detection results obtained by performing ROI detection on the image to be processed using N different detection operators.

[0006] Based on the N types of region of interest detection results, a target enhancement mode mask map and a target enhancement intensity mask map corresponding to the image to be processed are generated respectively;

[0007] The region of interest in the image to be processed is enhanced based on the target enhancement mode mask map and the target enhancement intensity mask map.

[0008] An apparatus for enhancing a region of interest includes: an acquisition module, a generation module, and an enhancement module;

[0009] The acquisition module is used to acquire N types of region of interest detection results for the image to be processed, where N is a positive integer greater than one. The N types of region of interest detection results include: detection results obtained by performing region of interest detection on the image to be processed using N different detection operators respectively.

[0010] The generation module is used to generate a target enhancement mode mask map and a target enhancement intensity mask map corresponding to the image to be processed, respectively, based on the N types of region of interest detection results.

[0011] The enhancement module is used to enhance the region of interest in the image to be processed based on the target enhancement mode mask map and the target enhancement intensity mask map.

[0012] An electronic device, comprising:

[0013] At least one processor; and

[0014] A memory communicatively connected to the at least one processor; wherein,

[0015] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described above.

[0016] A non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the methods described above.

[0017] A computer program product includes a computer program / instructions that, when executed by a processor, implement the method described above.

[0018] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0019] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0020] Figure 1 This is a flowchart of the first embodiment of the region of interest enhancement processing method described in this disclosure;

[0021] Figure 2 This is a flowchart of a second embodiment of the region of interest enhancement processing method described in this disclosure;

[0022] Figure 3 This is a schematic diagram of the composition structure of Embodiment 300 of the Region of Interest Enhancement Processing Apparatus described in this disclosure;

[0023] Figure 4 A schematic block diagram of an electronic device 400 that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0024] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0025] Furthermore, it should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0026] Figure 1 This is a flowchart of a first embodiment of the region of interest enhancement processing method described in this disclosure. Figure 1 As shown, the specific implementation methods are as follows.

[0027] In step 101, the detection results of N regions of interest (ROIs) of the image to be processed are obtained, where N is a positive integer greater than one. The N ROI detection results include the detection results obtained after performing ROI region detection on the image to be processed using N different detection operators.

[0028] In step 102, based on the N ROI region detection results, a target enhancement mode mask map and a target enhancement intensity mask map corresponding to the image to be processed are generated respectively.

[0029] In step 103, the ROI region in the image to be processed is enhanced according to the target enhancement mode mask map and the target enhancement intensity mask map.

[0030] Using the scheme described in the above method embodiment, a target enhancement mode mask map and a target enhancement intensity mask map can be generated based on the N ROI region detection results of the image to be processed. The ROI regions in the image to be processed can be enhanced based on the two mask maps. The image to be processed can be an image in a video, thereby improving the quality of the user's attention area and thus improving the subjective quality of the video.

[0031] The specific value of N can be determined according to actual needs. N different detection operators can be used to perform ROI region detection on the image to be processed, thus obtaining N ROI region detection results. For example, commonly used detection operators may include face detection operators, human body detection operators, caption detection operators, salient region detection operators, and contour detection operators, etc., and the corresponding ROI region detection results may be: face bounding boxes, human body bounding boxes, caption bounding boxes, salient region mask images, and contour points (i.e., the set of contour pixel positions), respectively. In practical applications, the number of ROI region detection results obtained may also be greater than N, and some of them, such as N, can be selected for processing according to the scheme described in this disclosure, while the remaining ROI region detection results can be used for other processing.

[0032] Based on the detection results of N types of ROI regions, a target enhancement mode mask map and a target enhancement intensity mask map corresponding to the image to be processed can be generated respectively.

[0033] In some embodiments of this disclosure, an initial enhancement mode mask image can be generated based on the image to be processed and a pre-set enhancement mode corresponding to the image to be processed. The initial enhancement mode mask image can then be optimized by assigning values ​​to the N ROI region detection results and the corresponding enhancement modes for each of the pre-set N ROI region detection results to obtain a target enhancement mode mask image. Additionally, an initial enhancement intensity mask image can be generated based on the image to be processed and a pre-set enhancement intensity corresponding to the image to be processed. The initial enhancement intensity mask image can then be optimized by assigning values ​​to the N ROI region detection results and the corresponding enhancement intensities for each of the pre-set N ROI region detection results to obtain a target enhancement intensity mask image. The pre-setting can be manually configured and can be adjusted at any time according to actual needs.

[0034] In some embodiments of this disclosure, the size of the initial enhancement mode mask image is the same as that of the image to be processed, and the value of each pixel in the initial enhancement mode mask image is a first initial value, which is a numerical representation of the enhancement mode corresponding to the image to be processed. Similarly, the size of the initial enhancement intensity mask image is the same as that of the image to be processed, and the value of each pixel in the initial enhancement intensity mask image is a second initial value, which is a numerical representation of the enhancement intensity corresponding to the image to be processed.

[0035] In other words, in the scheme described in this disclosure, the image to be processed (the whole frame image) will correspond to its own enhancement mode and enhancement intensity, and at the same time, the detection results of various ROI regions will also correspond to their respective enhancement modes and enhancement intensities.

[0036] Accordingly, two mask images can be constructed first according to the whole frame mode: an initial enhancement mode mask image and an initial enhancement intensity mask image. Both are the same size as the image to be processed. The value of each pixel in the initial enhancement mode mask image is equal to the numerical representation of the enhancement mode corresponding to the image to be processed, and the value of each pixel in the initial enhancement intensity mask image is equal to the numerical representation of the enhancement intensity corresponding to the image to be processed. Then, based on the N ROI region detection results and their corresponding enhancement modes and intensities, the initial enhancement mode mask image and initial enhancement intensity mask image can be optimized to obtain the desired target enhancement mode mask image and target enhancement intensity mask image.

[0037] Traditional methods typically process only a single Region of Interest (ROI), such as processing only faces, or merging faces, bodies, and captions into a single ROI before processing, and then configuring enhancement strategies for that ROI, resulting in poor enhancement effects. The solution described in this disclosure, however, enables multi-ROI enhancement based on mask images. It sets corresponding enhancement modes and intensities for the image to be processed and for different types of ROI detection results, thereby improving the enhancement effect and ultimately enhancing image quality.

[0038] Furthermore, in both the initial and target enhancement mode mask images, the value of each pixel can be represented by an 8-bit unsigned 8-bit integer (uint8_t). Each value corresponds to an enhancement mode, which can include only a specific enhancement operator, or it can include a specific enhancement operator and its related control information, such as the on / off state of sub-strategies within the enhancement operator. For example, a portion of the 8 bits can be used to represent the enhancement operator, while the remaining bits represent the related control information, offering great flexibility and convenience. The numerical representations corresponding to different enhancement modes can be determined according to actual needs. Enhancement operators can refer to operators such as Unsharp Mask (USM). Additionally, in both the initial and target enhancement intensity mask images, each pixel can be an 8-bit integer (int8_t) or a 16-bit integer (int16_t) value, corresponding to an integer-based enhancement intensity.

[0039] The initial augmentation mode mask map and the initial augmentation intensity mask map can be optimized by assigning values ​​to them respectively to obtain the target augmentation mode mask map and the target augmentation intensity mask map.

[0040] In some embodiments of this disclosure, the method for optimizing the initial enhancement mode mask image may include: sequentially traversing N types of ROI region detection results in ascending order of pre-set priority, and performing the following processing on each traversed ROI region detection result: taking the currently traversed ROI region detection result as the first detection result to be processed, determining the pixel point matching the first detection result to be processed from the initial enhancement mode mask image, and setting the value of the matching pixel point as the numerical representation of the enhancement mode corresponding to the first detection result to be processed; similarly, the method for optimizing the initial enhancement intensity mask image may include: sequentially traversing N types of ROI region detection results in ascending order of priority, and performing the following processing on each traversed ROI region detection result: taking the currently traversed ROI region detection result as the second detection result to be processed, determining the pixel point matching the second detection result to be processed from the initial enhancement intensity mask image, and setting the value of the matching pixel point as the numerical representation of the enhancement intensity corresponding to the second detection result to be processed.

[0041] There is no restriction on the priority order of the N types of ROI region detection results. For example, if the N types of ROI region detection results include: face bounding box, body bounding box, caption box, salient region mask image and contour point, then the order of priority from low to high can be: salient region mask image, contour point, caption box, body bounding box and face bounding box.

[0042] Taking the optimization of the initial enhancement mode mask image as an example, the detection result of the lowest priority ROI region can be taken as the first detection result to be processed. The pixel point that matches the first detection result to be processed can be determined from the initial enhancement mode mask image. Then, the value of the matching pixel point can be set as the numerical representation of the enhancement mode corresponding to the first detection result to be processed (at this time, the detection result of the lowest priority ROI region). Then, the detection result of the second lowest priority ROI region can be taken as the first detection result to be processed. The pixel point that matches the first detection result to be processed can be determined from the initial enhancement mode mask image. Then, the value of the matching pixel point can be set as the numerical representation of the enhancement mode corresponding to the first detection result to be processed (at this time, the detection result of the second lowest priority ROI region). This process is repeated until the detection result of the highest priority ROI region is also processed, and the target enhancement mode mask image is obtained.

[0043] Because the processing is performed in order of priority from low to high, the assignment results of high-priority regions will overwrite the assignment results of low-priority regions. In this way, even if a pixel matches multiple ROI region detection results, the final assignment will be based on the ROI region detection result with the highest priority, thus ensuring the accuracy of the processing results.

[0044] Furthermore, different types of ROI detection results can be represented in different forms. For example, face bounding boxes, body bounding boxes, and caption boxes are all rectangular bounding boxes. Contour points are sets of contour pixel locations, and salient region mask images are a single frame labeled with salient (represented by 1) and non-salient (represented by 0) regions. In traditional methods, merging different types of ROI detection results into a single ROI usually requires multiple processing steps. For example, face bounding boxes are processed first, then caption boxes, contour points, and salient region mask images are processed, then the entire image is processed, and finally the results are merged into a single image. This process is very complex, with high processing logic and computational complexity. Moreover, a pixel may have multiple attributes, such as being both a pixel on a face and a contour point pixel, making merging even more troublesome. The solution described in this disclosure only requires assigning values ​​according to the priority order of the detection results of different types of ROI regions to obtain the required target enhancement mode mask map. The whole process is simple and convenient to implement, thereby effectively simplifying the processing logic, reducing computational complexity, and improving processing efficiency.

[0045] In the process of determining the pixel that matches the first detection result from the initial enhanced mode mask image, the matching pixel can be determined in a corresponding way for different types of ROI region detection results. The methods mainly include the following methods one, two and three.

[0046] 1) Method 1

[0047] In response to determining that the first detection result to be processed is a detection result in the form of a rectangular box, the pixels in the rectangular region corresponding to the rectangular box in the initial enhanced mode mask image can be identified as matching pixels.

[0048] Detection results in the form of rectangular boxes can include: face boxes, body boxes, and text boxes, etc.

[0049] Taking a face bounding box as an example, assuming that the pixel with coordinates (12, 15) in the initial enhancement mode mask image is located in the rectangular area corresponding to the face bounding box, then this pixel can be identified as the matching pixel, and the value of the matching pixel can be set as the numerical representation of the enhancement mode corresponding to the face bounding box.

[0050] 2) Method Two

[0051] In response to determining that the first detection result to be processed is a salient region mask image, the corresponding pixel in the initial enhancement mode mask image of the pixel with a value of 1 in the salient region mask image can be determined as the matching pixel. The size of the salient region mask image is the same as that of the image to be processed, and the value of each pixel in the salient region mask image is 1 or 0, and the corresponding pixel is the pixel with the same coordinate position.

[0052] For example, if the pixel at coordinates (2, 4) in the salient region mask image has a value of 1, then the pixel at the same coordinate position in the initial enhancement mode mask image can be identified as the matching pixel. Furthermore, the value of the matching pixel can be set as the numerical representation of the enhancement mode corresponding to the salient region mask image.

[0053] 3) Method Three

[0054] In response to determining that the first detection result to be processed is a contour point, the pixels belonging to the contour point in the initial enhancement mode mask image can be identified as matching pixels, or the pixels belonging to the contour point in the initial enhancement mode mask image and the pixels adjacent to the pixels belonging to the contour point can be identified as matching pixels.

[0055] For example, if the pixel at coordinates (20, 20) in the initial enhancement mode mask image is a contour point, then this pixel can be identified as the matching pixel. Alternatively, this pixel and the pixels adjacent to it (up, down, left, right) can be identified as matching pixels, and the value of the matching pixel can be set as the numerical representation of the enhancement mode corresponding to the contour point.

[0056] As can be seen, in the above processing method, for different types of ROI region detection results, corresponding methods can be used to determine the matching pixels from the initial enhancement mode mask image, thereby improving the accuracy of the determination results, and further improving the accuracy of the subsequent target enhancement mode mask image.

[0057] Similarly, when determining the pixel that matches the second detection result from the initial enhancement intensity mask image, the matching pixel can be determined in a corresponding way for different types of ROI region detection results. The methods mainly include the following methods one, two and three.

[0058] 1) Method 1

[0059] In response to determining that the second detection result to be processed is a detection result in the form of a rectangular box, the pixels located in the rectangular region corresponding to the rectangular box in the initial enhancement intensity mask image can be identified as matching pixels.

[0060] Detection results in the form of rectangular boxes can include: face boxes, body boxes, and text boxes, etc.

[0061] 2) Method Two

[0062] In response to determining that the second detection result to be processed is a saliency region mask map, the corresponding pixel in the initial enhancement intensity mask map of the pixel with a value of 1 in the saliency region mask map can be determined as the matching pixel.

[0063] 3) Method Three

[0064] In response to determining that the second detection result to be processed is a contour point, the pixels belonging to the contour point in the initial enhancement intensity mask image can be identified as matching pixels, or the pixels belonging to the contour point in the initial enhancement intensity mask image and the pixels adjacent to the pixels belonging to the contour point can be identified as matching pixels.

[0065] Regardless of whether it is Method 1, Method 2, or Method 3, after determining the pixel that matches the second detection result from the initial enhancement intensity mask image, the value of the matching pixel can be set as the numerical representation of the enhancement intensity corresponding to the second detection result.

[0066] As can be seen, in the above processing method, for different types of ROI region detection results, corresponding methods can be used to determine the matching pixels from the initial enhancement intensity mask image, thereby improving the accuracy of the determination results, and further improving the accuracy of the subsequent target enhancement intensity mask image.

[0067] After obtaining the target enhancement mode mask map and the target enhancement intensity mask map respectively, the ROI region in the image to be processed can be enhanced based on these two mask maps.

[0068] In some embodiments of this disclosure, the following processing can be performed on each pixel in the image to be processed: obtaining the first value of the corresponding pixel in the target enhancement mode mask image and obtaining the second value of the corresponding pixel in the target enhancement intensity mask image, wherein the corresponding pixel is a pixel with the same coordinate position, and enhancing the pixel according to the enhancement mode corresponding to the first value and the enhancement intensity corresponding to the second value.

[0069] This allows for pixel-level processing, traversing each pixel in the image to be processed. When traversing each pixel, the values ​​of the corresponding pixels in the target enhancement mode mask image and the target enhancement intensity mask image can be obtained. This allows the enhancement mode and enhancement intensity of the pixel to be obtained, and the pixel can then be enhanced according to the obtained enhancement mode and enhancement intensity.

[0070] Using the above processing method, the enhancement processing of the image to be processed can be completed simply by using the target enhancement mode mask map and the target enhancement intensity mask map. The enhancement processing is decoupled from the ROI region detection results, thereby simplifying the processing logic and improving processing efficiency. Moreover, it can not only enhance the ROI region in the image to be processed, but also enhance the non-ROI region because the enhancement mode and enhancement intensity are also set for the image to be processed, thereby further improving the enhancement effect.

[0071] Based on the above introduction, Figure 2 This is a flowchart of a second embodiment of the region of interest enhancement processing method described in this disclosure. Figure 2 As shown, the specific implementation methods are as follows.

[0072] In step 201, N types of ROI region detection results of the image to be processed are obtained, where N is a positive integer greater than one. The N types of ROI region detection results include the detection results obtained after performing ROI region detection on the image to be processed using N different detection operators.

[0073] In step 202, an initial enhancement mode mask image is generated based on the image to be processed and the enhancement mode corresponding to the image to be processed that is preset. The size of the initial enhancement mode mask image is the same as that of the image to be processed, and the value of each pixel in the mask image is a first initial value, which is a numerical representation of the enhancement mode corresponding to the image to be processed.

[0074] In step 203, the N ROI region detection results are traversed sequentially in order of priority from low to high according to a pre-set priority, and the first ROI region detection result encountered is taken as the first detection result to be processed.

[0075] In step 204, the pixels that match the first detection result to be processed are determined from the initial enhancement mode mask image, and the values ​​of the matching pixels are set as the numerical representation of the enhancement mode corresponding to the first detection result to be processed.

[0076] In step 205, it is determined whether there are any untraversed ROI region detection results. If so, step 206 is executed; otherwise, step 207 is executed.

[0077] In step 206, the detection result of the next ROI region is taken as the first detection result to be processed, and then step 204 is repeated.

[0078] In step 207, the newly assigned initial augmented mode mask graph is determined as the target augmented mode mask graph, and then step 208 is executed.

[0079] In step 208, an initial enhancement intensity mask image is generated based on the image to be processed and the enhancement intensity corresponding to the image to be processed in a preset manner. The size of the initial enhancement intensity mask image is the same as that of the image to be processed, and the value of each pixel in the mask image is a second initial value, which is a numerical representation of the enhancement intensity corresponding to the image to be processed.

[0080] In step 209, the N ROI region detection results are traversed sequentially in order of priority from low to high according to a pre-set priority, and the first ROI region detection result encountered is taken as the second detection result to be processed.

[0081] In step 210, the pixels that match the second detection result to be processed are determined from the initial enhancement intensity mask image, and the values ​​of the matching pixels are set as the numerical representation of the enhancement intensity corresponding to the second detection result to be processed.

[0082] In step 211, it is determined whether there are any untraversed ROI region detection results. If so, step 212 is executed; otherwise, step 213 is executed.

[0083] In step 212, the detection result of the next ROI region is used as the second detection result to be processed, and then step 210 is repeated.

[0084] In step 213, the newly assigned initial enhancement intensity mask map is determined as the target enhancement intensity mask map, and then step 214 is executed.

[0085] In step 214, for each pixel in the image to be processed, the first value of the corresponding pixel in the target enhancement mode mask image is obtained, and the second value of the corresponding pixel in the target enhancement intensity mask image is obtained. The pixel is enhanced according to the enhancement mode corresponding to the first value and the enhancement intensity corresponding to the second value, and then the process ends.

[0086] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, because according to this disclosure, some steps can be performed in other orders or simultaneously. For example, steps 202-207 can be performed simultaneously with steps 208-213. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this disclosure. Furthermore, for parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0087] The above is an introduction to the method embodiments. The following describes the solution described in this disclosure further through device embodiments.

[0088] Figure 3 This is a schematic diagram of the structural composition of Embodiment 300 of the ROI region enhancement processing device described in this disclosure. Figure 3 As shown, it includes: an acquisition module 301, a generation module 302, and an enhancement module 303.

[0089] The acquisition module 301 is used to acquire N types of ROI region detection results of the image to be processed, where N is a positive integer greater than one. The N types of ROI region detection results include: the detection results obtained after performing ROI region detection on the image to be processed using N different detection operators.

[0090] The generation module 302 is used to generate a target enhancement mode mask map and a target enhancement intensity mask map corresponding to the image to be processed based on the N types of ROI region detection results.

[0091] The enhancement module 303 is used to enhance the ROI region in the image to be processed based on the target enhancement mode mask map and the target enhancement intensity mask map.

[0092] Using the scheme described in the above-described device embodiment, a target enhancement mode mask map and a target enhancement intensity mask map can be generated based on the N ROI region detection results of the image to be processed. The ROI regions in the image to be processed can be enhanced based on the two mask maps. The image to be processed can be an image in a video, thereby improving the quality of the user's attention area and thus improving the subjective quality of the video.

[0093] The specific value of N can be determined according to actual needs. N different detection operators can be used to perform ROI region detection on the image to be processed, thus obtaining N different ROI region detection results. For example, commonly used detection operators may include face detection operators, human body detection operators, caption detection operators, salient region detection operators, and contour detection operators, etc., and the corresponding ROI region detection results may be: face bounding boxes, human body bounding boxes, caption bounding boxes, salient region mask images, and contour points, respectively.

[0094] Based on the detection results of N types of ROI regions, the generation module 302 can generate the target enhancement mode mask map and the target enhancement intensity mask map corresponding to the image to be processed.

[0095] In some embodiments of this disclosure, the generation module 302 may include a first generation unit 3021 and a second generation unit 3022.

[0096] The first generation unit 3021 is used to generate an initial enhancement mode mask image based on the image to be processed and the enhancement mode corresponding to the pre-set image to be processed, and to assign and optimize the initial enhancement mode mask image according to the N types of ROI region detection results and the enhancement modes corresponding to the N types of ROI region detection results respectively, so as to obtain the target enhancement mode mask image.

[0097] The second generation unit 3022 is used to generate an initial enhancement intensity mask image based on the image to be processed and the enhancement intensity corresponding to the pre-set image to be processed, and to assign and optimize the initial enhancement intensity mask image based on the enhancement intensity corresponding to the N types of ROI region detection results and the pre-set N types of ROI region detection results respectively, so as to obtain the target enhancement intensity mask image.

[0098] In addition, in some embodiments of this disclosure, the size of the initial enhancement mode mask image is the same as that of the image to be processed, and the value of each pixel in the initial enhancement mode mask image is a first initial value, which is a numerical representation of the enhancement mode corresponding to the image to be processed. Similarly, the size of the initial enhancement intensity mask image is the same as that of the image to be processed, and the value of each pixel in the initial enhancement intensity mask image is a second initial value, which is a numerical representation of the enhancement intensity corresponding to the image to be processed.

[0099] In some embodiments of this disclosure, the method by which the first generation unit 3021 optimizes the initial enhancement mode mask image may include: sequentially traversing N types of ROI region detection results in ascending order of pre-set priority, and performing the following processing on each traversed ROI region detection result: taking the currently traversed ROI region detection result as the first detection result to be processed, determining the pixel point that matches the first detection result to be processed from the initial enhancement mode mask image, and setting the value of the matching pixel point as the numerical representation of the enhancement mode corresponding to the first detection result to be processed; similarly, the method by which the second generation unit 3022 optimizes the initial enhancement intensity mask image may include: sequentially traversing N types of ROI region detection results in ascending order of priority, and performing the following processing on each traversed ROI region detection result: taking the currently traversed ROI region detection result as the second detection result to be processed, determining the pixel point that matches the second detection result to be processed from the initial enhancement intensity mask image, and setting the value of the matching pixel point as the numerical representation of the enhancement intensity corresponding to the second detection result to be processed.

[0100] In addition, in some embodiments of this disclosure, the first generation unit 3021, in response to determining that the first detection result to be processed is a detection result in the form of a rectangular box, can determine the pixels in the rectangular region corresponding to the rectangular box in the initial enhancement mode mask image as matching pixels. Similarly, the second generation unit 3022, in response to determining that the second detection result to be processed is a detection result in the form of a rectangular box, can determine the pixels in the rectangular region corresponding to the rectangular box in the initial enhancement intensity mask image as matching pixels.

[0101] In some embodiments of this disclosure, in response to determining that the first detection result to be processed is a salient region mask image, the first generation unit 3021 can determine the corresponding pixel in the initial enhancement mode mask image of the pixel with a value of 1 in the salient region mask image as a matching pixel. The size of the salient region mask image is the same as that of the image to be processed, and the value of each pixel in the salient region mask image is 1 or 0, and the corresponding pixel is a pixel with the same coordinate position. Similarly, in response to determining that the second detection result to be processed is a salient region mask image, the second generation unit 3022 can determine the corresponding pixel in the initial enhancement intensity mask image of the pixel with a value of 1 in the salient region mask image as a matching pixel.

[0102] In some embodiments of this disclosure, in response to determining that the first detection result to be processed is a contour point, the first generation unit 3021 may determine the pixels belonging to the contour point in the initial enhancement mode mask image as matching pixels, or determine the pixels belonging to the contour point in the initial enhancement mode mask image and the pixels adjacent to the pixels belonging to the contour point as matching pixels. Similarly, in response to determining that the second detection result to be processed is a contour point, the second generation unit 3022 may determine the pixels belonging to the contour point in the initial enhancement intensity mask image as matching pixels, or determine the pixels belonging to the contour point in the initial enhancement intensity mask image and the pixels adjacent to the pixels belonging to the contour point as matching pixels.

[0103] After obtaining the target enhancement mode mask map and the target enhancement intensity mask map respectively, the enhancement module 303 can perform enhancement processing on the ROI region in the image to be processed based on these two mask maps.

[0104] In some embodiments of this disclosure, the enhancement module 303 may perform the following processing on each pixel in the image to be processed: obtain the first value of the corresponding pixel in the target enhancement mode mask image, and obtain the second value of the corresponding pixel in the target enhancement intensity mask image, wherein the corresponding pixel is a pixel with the same coordinate position, and perform enhancement processing on the pixel according to the enhancement mode corresponding to the first value and the enhancement intensity corresponding to the second value.

[0105] Figure 3 The specific workflow of the device embodiment shown can be found in the relevant descriptions in the foregoing method embodiments, and will not be repeated here.

[0106] The solutions described in this disclosure can be applied to the field of artificial intelligence, particularly in areas such as computer vision, image processing, and video encoders. Artificial intelligence is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. Artificial intelligence hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. Artificial intelligence software technologies mainly include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0107] The images and ROI region detection results in the embodiments described in this disclosure are not targeted at any specific user and do not reflect the personal information of any specific user. The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solutions of this disclosure all comply with relevant laws and regulations and do not violate public order and good morals.

[0108] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0109] Figure 4 A schematic block diagram of an electronic device 400 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0110] like Figure 4 As shown, device 400 includes a computing unit 401, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 402 or a computer program loaded from storage unit 408 into random access memory (RAM) 403. The RAM 403 may also store various programs and data required for the operation of device 400. The computing unit 401, ROM 402, and RAM 403 are interconnected via bus 404. An input / output (I / O) interface 405 is also connected to bus 404.

[0111] Multiple components in device 400 are connected to I / O interface 405, including: input unit 406, such as keyboard, mouse, etc.; output unit 407, such as various types of monitors, speakers, etc.; storage unit 408, such as disk, optical disk, etc.; and communication unit 409, such as network card, modem, wireless transceiver, etc. Communication unit 409 allows device 400 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0112] The computing unit 401 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. The computing unit 401 performs the various methods and processes described above, such as those described in this disclosure. For example, in some embodiments, the methods described in this disclosure can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 408. In some embodiments, part or all of the computer program can be loaded and / or installed on device 400 via ROM 402 and / or communication unit 409. When the computer program is loaded into RAM 403 and executed by the computing unit 401, one or more steps of the methods described in this disclosure can be performed. Alternatively, in other embodiments, the computing unit 401 may be configured to perform the methods described herein by any other suitable means (e.g., by means of firmware).

[0113] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard parts (ASSPs), systems-on-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0114] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0115] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory (EPROM), flash memory, optical fiber, compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0116] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0117] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0118] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0119] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0120] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for region of interest enhancement, comprising: Obtain N types of region of interest (ROI) detection results for the image to be processed, where N is a positive integer greater than one. The N types of ROI detection results include: detection results obtained by performing ROI detection on the image to be processed using N different detection operators. Based on the N types of region of interest (ROI) detection results, a target enhancement mode mask and a target enhancement intensity mask corresponding to the image to be processed are generated respectively. Generating the target enhancement mode mask includes: generating an initial enhancement mode mask based on the image to be processed and a pre-set enhancement mode corresponding to the image to be processed; sequentially traversing the N types of ROI detection results in ascending order of pre-set priority; and for each ROI detection result encountered, determining it as the first detection result to be processed; identifying pixels matching the first detection result from the initial enhancement mode mask; and setting the value of the matching pixel as the numerical representation of the enhancement mode corresponding to the first detection result to be processed. Identifying the matching pixel includes: in response to determining that the first detection result to be processed is a rectangular bounding box, setting the value of the matching pixel as the numerical representation of the enhancement mode corresponding to the first detection result to be processed. In the initial enhancement mode mask image, the pixels located in the rectangular region corresponding to the rectangle are determined as the matching pixels. In response to determining that the first detection result to be processed is a salient region mask image, the corresponding pixels in the initial enhancement mode mask image for pixels with a value of 1 in the salient region mask image are determined as the matching pixels. The size of the salient region mask image is the same as that of the image to be processed, and the value of each pixel in the salient region mask image is 1 or 0. The corresponding pixels are pixels with the same coordinate position. In response to determining that the first detection result to be processed is a contour point, the pixels in the initial enhancement mode mask image belonging to the contour point are determined as the matching pixels. Alternatively, the pixels in the initial enhancement mode mask image belonging to the contour point and the pixels adjacent to the pixels belonging to the contour point are determined as the matching pixels. The region of interest in the image to be processed is enhanced based on the target enhancement mode mask map and the target enhancement intensity mask map.

2. The method according to claim 1, wherein, Generating the target enhancement intensity mask map includes: An initial enhancement intensity mask is generated based on the image to be processed and the enhancement intensity corresponding to the image to be processed in a preset manner. The initial enhancement intensity mask is then optimized by assigning values ​​to it based on the N types of region of interest detection results and the enhancement intensities corresponding to the N types of region of interest detection results in a preset manner, so as to obtain the target enhancement intensity mask.

3. The method according to claim 2, wherein, The initial enhancement mode mask image is the same size as the image to be processed, and the value of each pixel in the initial enhancement mode mask image is a first initial value, which is a numerical representation of the enhancement mode corresponding to the image to be processed. The initial enhancement intensity mask is the same size as the image to be processed, and the value of each pixel in the initial enhancement intensity mask is a second initial value, which is a numerical representation of the enhancement intensity of the image to be processed.

4. The method according to claim 3, wherein, The optimization of the initial enhanced intensity mask image includes: The N types of region of interest detection results are traversed sequentially in order of priority from low to high. For each region of interest detection result traversed, the following processing is performed: the currently traversed region of interest detection result is taken as the second detection result to be processed. The pixel point that matches the second detection result to be processed is determined from the initial enhancement intensity mask map, and the value of the matching pixel point is set as the numerical representation of the enhancement intensity corresponding to the second detection result to be processed.

5. The method according to claim 4, wherein, In response to determining that the second detection result to be processed is a detection result in the form of a rectangular box, determining the pixel point that matches the second detection result to be processed from the initial enhancement intensity mask image includes: determining the pixel point located in the rectangular region corresponding to the rectangular box in the initial enhancement intensity mask image as the matching pixel point.

6. The method according to claim 4, wherein, In response to determining that the second detection result to be processed is the saliency region mask image, determining the pixel point that matches the second detection result to be processed from the initial enhancement intensity mask image includes: determining the corresponding pixel point in the initial enhancement intensity mask image of the pixel point with a value of 1 in the saliency region mask image as the matching pixel point.

7. The method according to claim 4, wherein, In response to determining that the second detection result to be processed is the contour point, determining the pixel point that matches the second detection result to be processed from the initial enhancement intensity mask includes: determining the pixel point belonging to the contour point in the initial enhancement intensity mask as the matching pixel point, or determining the pixel point belonging to the contour point and the pixel point adjacent to the pixel point belonging to the contour point in the initial enhancement intensity mask as the matching pixel point.

8. The method according to any one of claims 4 to 7, wherein, The enhancement processing of the region of interest in the image to be processed includes: For each pixel in the image to be processed, the following processing is performed: Obtain the first value of the corresponding pixel in the target enhancement mode mask image, and obtain the second value of the corresponding pixel in the target enhancement intensity mask image, wherein the corresponding pixel is a pixel with the same coordinate position; The pixel is enhanced according to the enhancement mode corresponding to the first value and the enhancement intensity corresponding to the second value.

9. A region-of-interest enhancement processing apparatus, comprising: Acquisition module, generation module, and enhancement module; The acquisition module is used to acquire N types of region of interest detection results for the image to be processed, where N is a positive integer greater than one. The N types of region of interest detection results include: detection results obtained by performing region of interest detection on the image to be processed using N different detection operators respectively. The generation module is configured to generate a target enhancement mode mask and a target enhancement intensity mask corresponding to the image to be processed based on the N types of region of interest (ROI) detection results. Generating the target enhancement mode mask includes: generating an initial enhancement mode mask based on the image to be processed and a pre-set enhancement mode corresponding to the image to be processed; sequentially traversing the N types of ROI detection results in ascending order of pre-set priority; and for each ROI detection result encountered, determining it as a first detection result to be processed; identifying pixels matching the first detection result from the initial enhancement mode mask; and setting the value of the matching pixel as the numerical representation of the enhancement mode corresponding to the first detection result to be processed. Identifying the matching pixel includes: responding to the determination that the first detection result to be processed is a rectangular bounding box. If the pixel in the rectangular region corresponding to the rectangle in the initial enhancement mode mask is determined as the matching pixel, in response to determining that the first detection result to be processed is a salient region mask, the corresponding pixel in the initial enhancement mode mask for the pixel with a value of 1 in the salient region mask is determined as the matching pixel. The size of the salient region mask is the same as that of the image to be processed, and the value of each pixel in the salient region mask is 1 or 0. The corresponding pixel is a pixel with the same coordinate position. In response to determining that the first detection result to be processed is a contour point, the pixel in the initial enhancement mode mask belonging to the contour point is determined as the matching pixel, or the pixel in the initial enhancement mode mask belonging to the contour point and the pixel adjacent to the pixel belonging to the contour point are determined as the matching pixel. The enhancement module is used to enhance the region of interest in the image to be processed based on the target enhancement mode mask map and the target enhancement intensity mask map.

10. The apparatus according to claim 9, wherein, The generation module generates an initial enhancement intensity mask based on the image to be processed and the enhancement intensity corresponding to the image to be processed in a preset manner. Based on the N types of region of interest detection results and the enhancement intensity corresponding to the N types of region of interest detection results in a preset manner, the initial enhancement intensity mask is optimized by assigning values ​​to it to obtain the target enhancement intensity mask.

11. The apparatus according to claim 10, wherein, The initial enhancement mode mask image is the same size as the image to be processed, and the value of each pixel in the initial enhancement mode mask image is a first initial value, which is a numerical representation of the enhancement mode corresponding to the image to be processed. The initial enhancement intensity mask is the same size as the image to be processed, and the value of each pixel in the initial enhancement intensity mask is a second initial value, which is a numerical representation of the enhancement intensity of the image to be processed.

12. The apparatus according to claim 11, wherein, The generation module sequentially traverses the N types of region of interest detection results in ascending order of priority, and performs the following processing on each region of interest detection result: the currently traversed region of interest detection result is taken as the second detection result to be processed, the pixel point that matches the second detection result to be processed is determined from the initial enhancement intensity mask map, and the value of the matching pixel point is set as the numerical representation of the enhancement intensity corresponding to the second detection result to be processed.

13. The apparatus according to claim 12, wherein, In response to determining that the second detection result to be processed is a detection result in the form of a rectangular box, the generation module determines the pixels in the rectangular region corresponding to the rectangular box in the initial enhancement intensity mask image as the matching pixels.

14. The apparatus according to claim 12, wherein, In response to determining that the second detection result to be processed is the saliency region mask image, the generation module determines the corresponding pixel in the initial enhancement intensity mask image of the pixel with a value of 1 in the saliency region mask image as the matching pixel.

15. The apparatus according to claim 12, wherein, In response to determining that the second detection result to be processed is the contour point, the generation module determines the pixels belonging to the contour point in the initial enhancement intensity mask as the matching pixels, or determines the pixels belonging to the contour point and the pixels adjacent to the pixels belonging to the contour point in the initial enhancement intensity mask as the matching pixels.

16. The apparatus according to any one of claims 12 to 15, wherein, The enhancement module performs the following processing on each pixel in the image to be processed: obtaining a first value of the corresponding pixel in the target enhancement mode mask image and obtaining a second value of the corresponding pixel in the target enhancement intensity mask image, wherein the corresponding pixel is a pixel with the same coordinate position, and enhancing the pixel according to the enhancement mode corresponding to the first value and the enhancement intensity corresponding to the second value.

17. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-8.

19. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the method of any one of claims 1-8.

Images