Code rate allocation method and device, electronic equipment and storage medium

Abstract

The invention provides a code rate allocation method and device, electronic equipment and a storage medium, and relates to the field of artificial intelligence such as computer vision, image processing and video encoders. The method can comprise the steps that N region-of-interest detection results of a to-be-processed image are obtained, N is a positive integer larger than one, and the N region-of-interest detection results comprise detection results obtained after region-of-interest detection is conducted on the to-be-processed image through N different detection operators; generating a target block quantization parameter offset mask graph corresponding to the to-be-processed image according to the N types of region-of-interest detection results; and performing code rate allocation on each image block in the to-be-processed image according to the target block quantization parameter offset mask graph.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a bit rate allocation method, device, electronic device and storage medium in the fields of computer vision, image processing and video encoder. Background Art

[0002] At present, video has become an indispensable part of people's lives. People can conveniently watch various small videos, short videos, live broadcasts, movies or TV series through various applications (APPs). For video manufacturers, how to improve the subjective quality of videos is an urgent problem to be solved. Summary of the invention

[0003] The present disclosure provides a code rate allocation method, device, electronic device and storage medium.

[0004] A code rate allocation method, comprising:

[0005] Obtaining N types of interest region detection results of the image to be processed, where N is a positive integer greater than one, and the N types of interest region detection results include: detection results obtained by performing interest region detection on the image to be processed using N different detection operators respectively;

[0006] Generating a target block quantization parameter offset mask map corresponding to the image to be processed according to the N types of region of interest detection results;

[0007] The code rate is allocated to each image block in the image to be processed according to the target block quantization parameter offset mask map.

[0008] A code rate allocation device comprises: an acquisition module, a generation module and an allocation module;

[0009] The acquisition module is used to acquire N types of interest region detection results of the image to be processed, where N is a positive integer greater than one, and the N types of interest region detection results include: detection results obtained by performing interest region detection on the image to be processed using N different detection operators respectively;

[0010] The generating module is used to generate a target block quantization parameter offset mask map corresponding to the image to be processed according to the N types of region of interest detection results;

[0011] The allocation module is used to allocate code rates to each image block in the image to be processed according to the target block quantization parameter offset mask map.

[0012] An electronic device, comprising:

[0013] at least one processor; and

[0014] a memory communicatively connected to the at least one processor; wherein,

[0015] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the method described above.

[0016] A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the method as described above.

[0017] A computer program product comprises a computer program / instruction, wherein the computer program / instruction implements the above method when executed by a processor.

[0018] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0020] Figure 1 This is a flow chart of the first embodiment of the code rate allocation method described in the present disclosure;

[0021] Figure 2 This is a flow chart of the second embodiment of the code rate allocation method described in the present disclosure;

[0022] Figure 3 This is a schematic diagram of the overall implementation process of the code rate allocation method disclosed in the present invention;

[0023] Figure 4 It is a schematic diagram of the composition structure of the bit rate allocation device embodiment 400 of the present disclosure;

[0024] Figure 5 A schematic block diagram of an electronic device 500 that can be used to implement an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0025] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0026] In addition, it should be understood that the term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0027] Figure 1 FIG. 1 is a flow chart of the first embodiment of the code rate allocation method described in the present disclosure. Figure 1 As shown, the following specific implementation methods are included.

[0028] In step 101, N types of region of interest (ROI) detection results of the image to be processed are obtained, where N is a positive integer greater than one, and the N types of ROI area detection results include: detection results obtained by performing ROI area detection on the image to be processed using N different detection operators.

[0029] In step 102, a target block quantization parameter (QP, QuantizerParameter) offset mask map corresponding to the image to be processed is generated according to the N types of ROI region detection results.

[0030] In step 103, code rate allocation is performed on each image block in the to-be-processed image according to the target block QP offset mask map.

[0031] By adopting the scheme described in the above method embodiment, a target block QP offset mask map can be generated according to the N types of ROI area detection results of the image to be processed, and the bit rate allocation can be performed on each image block in the image to be processed according to the target block QP offset mask map. The image to be processed can be the original image in the video, or the image after a certain optimization processing is performed on the original image in the video, thereby improving the rationality of the bit rate allocation, and then improving the image quality, and improving the subjective quality of the video and the user's viewing experience.

[0032] The specific value of N can be determined according to actual needs. N different detection operators can be used to perform ROI area detection on the image to be processed, thereby obtaining N ROI area detection results. For example, commonly used detection operators may include face detection operators, human body detection operators, subtitle detection operators, salient area detection operators, and contour detection operators, etc. The corresponding ROI area detection results may be: face frame, human body frame, subtitle frame, salient area mask map, and contour point (i.e., contour pixel position set). In practical applications, the number of ROI area detection results obtained may also be greater than N, and some of them, such as N, may be selected for processing according to the scheme described in the present disclosure, and the remaining ROI area detection results may be used for other processing.

[0033] According to the N types of ROI region detection results, a QP offset mask map of the target block corresponding to the image to be processed can be generated.

[0034] In some embodiments of the present disclosure, an initial block QP offset mask map may be first generated based on the image to be processed, and then the initial block QP offset mask map may be assigned and optimized based on N types of ROI area detection results and the QP offset values ​​corresponding to the N types of ROI area detection results, and then the target block QP offset mask map may be determined based on the assignment optimization result.

[0035] In some embodiments of the present disclosure, the initial block QP offset mask image may be of size M*N, where the value of M is equal to the ratio of the length of the image to be processed to L, and the value of N is equal to the ratio of the width of the image to be processed to L. In addition, the value of each pixel point in the initial block QP offset mask image may be 0, and each pixel point corresponds to an image block of size L*L in the image to be processed, and there is no overlap between any two image blocks.

[0036] For a frame of image, the bit rate allocation is done based on blocks (image blocks), and the bit rate allocation is achieved through QP. The larger the QP value, the lower the encoding bit rate. Conversely, the smaller the QP value, the higher the encoding bit rate.

[0037] The size of a single image block (i.e., the value of L) can be determined according to actual needs, such as 16*16 or 8*8. The following takes 16*16 as an example. Accordingly, the length and width of the image to be processed need to be integer multiples of 16. If they are not integer multiples of 16, they can be adjusted through preprocessing. Assuming that the image to be processed is 1600*1600 in size, the image to be processed can be divided into 100*100 image blocks. In this case, the size of the initial block QP offset mask image also needs to be 100*100.

[0038] The value of each pixel in the initial block QP offset mask is 0 (no QP adjustment), and each pixel corresponds to an image block in the image to be processed. After the initial block QP offset mask is obtained, the initial block QP offset mask can be further optimized according to the N ROI area detection results and the QP offset values ​​corresponding to the N ROI area detection results, and then the target block QP offset mask can be determined according to the optimization result.

[0039] That is to say, in the solution described in the present disclosure, various ROI region detection results may correspond to respective QP offset values, and the specific values ​​may be determined according to actual needs.

[0040] In addition, in the traditional method, usually only a single ROI area is processed, such as only a face is processed, or the face, body and subtitles are combined into one ROI area for processing, and then different bit rate allocation strategies are configured for the ROI area and the non-ROI area, which achieves poor results. The scheme described in the present disclosure can realize bit rate allocation for multiple ROI areas based on mask images, thereby improving the bit rate allocation effect, thereby improving the subjective quality of the video, etc.

[0041] In addition, no matter it is the initial block QP offset mask map or the target block QP offset mask map, the value of each pixel therein can be represented by an 8-bit integer (int8_t).

[0042] For the initial block QP offset mask map, the assignment optimization can be performed according to the N types of ROI region detection results and the QP offset values ​​corresponding to the N types of ROI region detection results.

[0043] In some embodiments of the present disclosure, the method for performing assignment optimization may include: traversing N types of ROI area detection results in order from low to high according to a preset priority, and performing the following processing for each traversed ROI area detection result: using the currently traversed ROI area detection result as the detection result to be processed, determining the pixel points that match the detection result to be processed from the initial block QP offset mask map, and setting the value of the matching pixel points to the QP offset value corresponding to the detection result to be processed.

[0044] There is no restriction on the priority order of N ROI region detection results. For example, assuming that the N ROI region detection results include: face frame, body frame, subtitle frame, salient region mask map and contour point, then the order from low to high priority can be: salient region mask map, contour point, subtitle frame, body frame and face frame.

[0045] Specifically, the ROI area detection result with the lowest priority can be first used as the detection result to be processed, and the pixel points matching the detection result to be processed can be determined from the initial block QP offset mask map, and then the values ​​of the matching pixel points can be set to the QP offset value corresponding to the detection result to be processed (in this case, the ROI area detection result with the lowest priority). Then, the ROI area detection result with the second lowest priority can be used as the detection result to be processed, and the pixel points matching the detection result to be processed can be determined from the initial block QP offset mask map, and then the values ​​of the matching pixel points can be set to the QP offset value corresponding to the detection result to be processed (in this case, the ROI area detection result with the second lowest priority), and so on, until the above processing is completed for the ROI area detection result with the highest priority, and the assignment optimization result is obtained.

[0046] Since the processing is performed in order from low to high priority, the high-priority assignment result will overwrite the low-priority assignment result. In this way, even if a pixel point matches multiple ROI area detection results, it will eventually be assigned according to the ROI area detection result with the highest priority, thereby ensuring the accuracy of the processing results.

[0047] In addition, different types of ROI region detection results can be expressed in different forms, such as face frame, human body frame and subtitle frame are all detection results in the form of rectangular frames, contour points are contour pixel position sets, and significant region mask images are mask images with significant (represented by 1) and non-significant (represented by 0) marked in one frame. In the traditional method, when merging different types of ROI region detection results into one ROI region, it is usually necessary to perform multiple processing, such as first processing the face frame, then processing the subtitle frame, contour points, and significant region mask images, and then processing the entire image, and finally merging the various processing results into one image, etc. The process is very complicated, and the processing logic and computational complexity are very high. Moreover, a certain pixel point may have multiple attributes, such as being both a pixel on the face and a contour point pixel, and merging them will also be very troublesome. In the scheme described in the present disclosure, it is only necessary to assign values ​​according to the priority order of different types of ROI region detection results to obtain the required target result. The whole process is simple and convenient to implement, thereby effectively simplifying the processing logic, reducing the computational complexity, and improving the processing efficiency.

[0048] Among them, when determining the pixel points matching the detection result to be processed from the initial block QP offset mask map, for different types of ROI area detection results, corresponding methods can be used to determine the matching pixel points, which mainly include the following method 1, method 2 and method 3.

[0049] 1) Method 1

[0050] In response to determining that the detection result to be processed is a detection result in the form of a rectangular frame, a pixel point in the initial block QP offset mask image that meets the following requirements can be determined as a matching pixel point: the image block corresponding to the pixel point overlaps with the rectangular frame.

[0051] The detection results in the form of rectangular frames may include: face frames, body frames, and subtitle frames.

[0052] Taking the face frame as an example, assuming that the image block corresponding to the pixel with coordinate position (12, 15) in the initial block QP offset mask is completely or partially located within a face frame, then the pixel can be determined as a matching pixel, and the value of the matching pixel can be set as the QP offset value corresponding to the face frame.

[0053] 2) Method 2

[0054] In response to determining that the detection result to be processed is a salient area mask map, pixel points in the initial block QP offset mask map that meet the following requirements can be determined as matching pixel points: the proportion of the number of first-category pixel points in the image block corresponding to the pixel point in the total number of pixel points in the image block corresponding to the pixel point is greater than a first threshold, the first-category pixel points are pixel points whose values ​​in the salient area mask map are 1, the size of the salient area mask map is the same as that of the image to be processed, and the value of each pixel point in the salient area mask map is 1 or 0, respectively.

[0055] The specific value of the first threshold can be determined according to actual needs, such as 50%. Assume that the image block corresponding to the pixel point with coordinate position (2, 4) in the initial block QP offset mask map is image block x, the total number of pixels in image block x is 16*16=256, and the number of pixels in image block x with a value of 1 in the salient area mask map is 180. Since 180 accounts for about 70% of 256, which is greater than the first threshold, the pixel point can be determined as a matching pixel point, and the value of the matching pixel point can be set as the QP offset value corresponding to the salient area mask map.

[0056] 3) Method 3

[0057] In response to determining that the detection result to be processed is a contour point, a pixel point in the initial block QP offset mask image that meets the following requirements can be determined as a matching pixel point: the proportion of the number of second-category pixel points in the image block corresponding to the pixel point in the total number of pixel points in the image block corresponding to the pixel point is greater than a second threshold, and the second-category pixel point is a pixel point that belongs to a contour point.

[0058] The specific value of the second threshold value can be determined according to actual needs, such as 10%. Assume that the image block corresponding to the pixel point with coordinate position (10, 10) in the initial block QP offset mask is image block y, the total number of pixels in image block y is 16*16=256, and the number of pixels belonging to contour points in image block y is 50. Since 50 accounts for about 20% of 256, which is greater than the second threshold value, the pixel point can be determined as a matching pixel point, and the value of the matching pixel point can be set as the QP offset value corresponding to the contour point.

[0059] It can be seen that in the above processing method, for different types of ROI area detection results, corresponding methods can be used to determine matching pixels from the initial block QP offset mask map, thereby improving the accuracy of the determination results, and further improving the accuracy of the assignment optimization results.

[0060] According to the assignment optimization result, a target block QP offset mask map may be determined. In some embodiments of the present disclosure, the assignment optimization result may be directly determined as the target block QP offset mask map, or the assignment optimization result may be determined as the intermediate block QP offset mask map, and the average value of each pixel point in the intermediate block QP offset mask map may be obtained, and then the value of each pixel point in the intermediate block QP offset mask map may be added to the average value to obtain the target block QP offset mask map.

[0061] That is to say, after the initial block QP offset mask map is assigned and optimized, the assignment optimization result can be directly determined as the target block QP offset mask map, or further optimization can be performed based on the assignment optimization result. For ease of expression, the assignment optimization result is called the intermediate block QP offset mask map, and the average value of each pixel point in the intermediate block QP offset mask map can be obtained, which is assumed to be avg_deltaQP (usually a negative number), and the value of each pixel point in the intermediate block QP offset mask map can be added to avg_deltaQP respectively, so as to obtain the required target block QP offset mask map.

[0062] Which of the above methods is used can be determined according to actual needs, which is very flexible and convenient. However, preferably, the latter method can be used. By using this method, the sum of the offset values ​​of all image blocks can be made 0, avoiding a significant impact on the overall bit rate. From the implementation point of view, only some extra bit rates are allocated to the ROI area to improve the image and video quality perceived by the human eye, and the bit rates of the non-ROI area are correspondingly reduced, thereby maintaining the overall bit rate as much as possible and reducing resource consumption.

[0063] Furthermore, the code rate may be allocated to each image block in the image to be processed according to the obtained target block QP offset mask map.

[0064] In some embodiments of the present disclosure, the following processing may be performed for any image block in the image to be processed: an initial block QP of the image block determined according to an adaptive quantization (AQ) algorithm is obtained, and the values ​​of the corresponding pixel points of the image block in the target block QP offset mask map are superimposed on the initial block QP to obtain a bit rate allocation result for the image block.

[0065] According to the traditional AQ algorithm, the initial block QP of each image block can be obtained respectively. On this basis, in the scheme described in the present invention, an additional offset value is added to each image block according to the target block QP offset mask map, thereby realizing a bit rate allocation strategy for multiple ROI areas. While maintaining the overall bit rate unchanged as much as possible, more bit rate is allocated to the ROI area that the human eye pays attention to, thereby improving the quality of images and videos.

[0066] Based on the above introduction, Figure 2 FIG. 1 is a flow chart of the second embodiment of the code rate allocation method described in the present disclosure. Figure 2 As shown, the following specific implementation methods are included.

[0067] In step 201, N ROI detection results of the image to be processed are obtained, where N is a positive integer greater than one, and the N ROI region detection results include: detection results obtained by performing ROI region detection on the image to be processed using N different detection operators.

[0068] In step 202, an initial block QP offset mask map is generated according to the image to be processed. The initial block QP offset mask map is of M*N size, where the value of M is equal to the ratio of the length of the image to be processed to L, the value of N is equal to the ratio of the width of the image to be processed to L, and the value of each pixel in the initial block QP offset mask map is 0, and each pixel corresponds to an image block of L*L size in the image to be processed, and there is no overlap between any two image blocks.

[0069] In step 203, N types of ROI region detection results are traversed in order from low to high according to the preset priorities, and the first traversed ROI region detection result is used as the detection result to be processed.

[0070] In step 204, pixel points that match the detection result to be processed are determined from the initial block QP offset mask map, and the values ​​of the matching pixel points are set as the QP offset values ​​corresponding to the detection result to be processed.

[0071] In step 205 , it is determined whether there is an untraversed ROI region detection result, if so, step 206 is executed, otherwise, step 207 is executed.

[0072] In step 206, the next ROI region detection result is used as the detection result to be processed, and then step 204 is repeated.

[0073] In step 207, the latest assigned initial block QP offset mask map is determined as the intermediate block QP offset mask map.

[0074] In step 208, the mean value of the values ​​of each pixel point in the intermediate block QP offset mask map is obtained, and the mean value is added to the value of each pixel point in the intermediate block QP offset mask map to obtain the target block QP offset mask map.

[0075] In step 209, for any image block in the image to be processed, the following processing is performed respectively: the initial block QP of the image block determined according to the AQ algorithm is obtained, and the values ​​of the corresponding pixel points of the image block in the target block QP offset mask map are superimposed on the initial block QP to obtain the bit rate allocation result of the image block, and then the process ends.

[0076] Figure 3 FIG. 1 is a schematic diagram of the overall implementation process of the code rate allocation method described in the present disclosure. Figure 3 As shown in FIG. 1 , for the image to be processed, N different detection operators can be used to perform ROI region detection on it, thereby obtaining N ROI region detection results, such as: face frame, body frame, subtitle frame, salient region mask map and contour points. Figure 3 As shown, an initial block QP offset mask map can be generated, and the initial block QP offset mask map can be assigned and optimized, wherein the initial block QP offset mask map is M*N in size, the value of M is equal to the ratio of the length of the image to be processed to L, the value of N is equal to the ratio of the width of the image to be processed to L, and in addition, the value of each pixel in the initial block QP offset mask map can be 0, and each pixel corresponds to an image block of L*L in the image to be processed, there is no overlap between any two image blocks, and according to the N kinds of ROI area detection results and The QP offset values ​​corresponding to the N ROI area detection results are respectively used to optimize the assignment of the initial block QP offset mask map. For example, the N ROI area detection results can be traversed in order from low to high according to the preset priority, and the following processing can be performed for each traversed ROI area detection result: the currently traversed ROI area detection result is used as the detection result to be processed, and the pixel points that match the detection result to be processed are determined from the initial block QP offset mask map, and the values ​​of the matching pixels are set as the QP offset values ​​corresponding to the detection result to be processed. Figure 3 As shown, the obtained assignment optimization result can be further optimized, such as using the assignment optimization result as the intermediate block QP offset mask map, and obtaining the average value of each pixel point in the intermediate block QP offset mask map, and then adding the average value to the value of each pixel point in the intermediate block QP offset mask map to obtain the target block QP offset mask map. Figure 3As shown, further, the bit rate allocation can be performed on each image block in the image to be processed according to the target block QP offset mask map, so as to obtain the final processing result. For example, the following processing can be performed for any image block in the image to be processed: the initial block QP of the image block determined according to the AQ algorithm is obtained, and the values ​​of the corresponding pixel points of the image block in the target block QP offset mask map are superimposed on the initial block QP to obtain the bit rate allocation result of the image block.

[0077] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present disclosure is not limited by the described order of actions, because according to the present disclosure, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present disclosure. In addition, for parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0078] The above is an introduction to the method embodiment. The following is a further explanation of the scheme disclosed in the present invention through an apparatus embodiment.

[0079] Figure 4 FIG. 4 is a schematic diagram of the structure of the bit rate allocation device embodiment 400 described in the present disclosure. Figure 4 As shown, it includes: an acquisition module 401, a generation module 402 and an allocation module 403.

[0080] The acquisition module 401 is used to acquire N ROI region detection results of the image to be processed, where N is a positive integer greater than one, and the N ROI region detection results include: detection results obtained by performing ROI region detection on the image to be processed using N different detection operators.

[0081] The generating module 402 is used to generate a QP offset mask map of a target block corresponding to the image to be processed according to the N types of ROI region detection results.

[0082] The allocation module 403 is used to allocate code rates to each image block in the image to be processed according to the target block QP offset mask map.

[0083] By adopting the scheme described in the above-mentioned device embodiment, a target block QP offset mask map can be generated according to the N types of ROI area detection results of the image to be processed, and the bit rate allocation can be performed on each image block in the image to be processed according to the target block QP offset mask map. The image to be processed can be the original image in the video, or the image after a certain optimization processing is performed on the original image in the video, thereby improving the rationality of the bit rate allocation, and then improving the image quality, and improving the subjective quality of the video and the user's viewing experience.

[0084] N different detection operators can be used to perform ROI region detection on the image to be processed, thereby obtaining N ROI region detection results. For example, commonly used detection operators may include face detection operators, human body detection operators, subtitle detection operators, salient region detection operators, and contour detection operators, and the corresponding ROI region detection results may be: face frame, human body frame, subtitle frame, salient region mask map, and contour points.

[0085] According to the N types of ROI region detection results, the generation module 402 may generate a QP offset mask map of the target block corresponding to the image to be processed.

[0086] In some embodiments of the present disclosure, the generation module 402 may first generate an initial block QP offset mask map according to the image to be processed, and then may perform assignment optimization on the initial block QP offset mask map according to N types of ROI area detection results and the QP offset values ​​corresponding to the N types of ROI area detection results, and then may determine the target block QP offset mask map according to the assignment optimization result.

[0087] In some embodiments of the present disclosure, the initial block QP offset mask image may be of size M*N, where the value of M is equal to the ratio of the length of the image to be processed to L, and the value of N is equal to the ratio of the width of the image to be processed to L. In addition, the value of each pixel in the initial block QP offset mask image is 0, and each pixel corresponds to an image block of size L*L in the image to be processed, and there is no overlap between any two image blocks.

[0088] In some embodiments of the present disclosure, the generation module 402 may perform value assignment optimization in a manner that may include: traversing N types of ROI area detection results in order from low to high according to a preset priority, and performing the following processing for each traversed ROI area detection result: using the currently traversed ROI area detection result as a detection result to be processed, determining pixel points that match the detection result to be processed from the initial block QP offset mask map, and setting the value of the matching pixel points to the QP offset value corresponding to the detection result to be processed.

[0089] When determining the pixel points that match the detection result to be processed from the initial block QP offset mask image, the matching pixel points can be determined in corresponding ways for different types of ROI area detection results.

[0090] In some embodiments of the present disclosure, in response to determining that the detection result to be processed is a detection result in the form of a rectangular frame, the generation module 402 may determine the pixel points in the initial block QP offset mask image that meet the following requirements as matching pixel points: the image block corresponding to the pixel point overlaps with the rectangular frame.

[0091] In some embodiments of the present disclosure, in response to determining that the detection result to be processed is a salient area mask map, the generation module 402 may determine the pixel points in the initial block QP offset mask map that meet the following requirements as matching pixel points: the number of first-category pixel points in the image block corresponding to the pixel point accounts for a proportion of the total number of pixel points in the image block corresponding to the pixel point that is greater than a first threshold, the first-category pixel points are pixel points whose value in the salient area mask map is 1, the size of the salient area mask map is the same as that of the image to be processed, and the value of each pixel point in the salient area mask map is 1 or 0, respectively.

[0092] In some embodiments of the present disclosure, in response to determining that the detection result to be processed is a contour point, the generation module 402 may determine the pixel points in the initial block QP offset mask image that meet the following requirements as matching pixel points: the proportion of the second type of pixel points in the image block corresponding to the pixel point in the total number of pixel points in the image block corresponding to the pixel point is greater than a second threshold, and the second type of pixel points are pixel points that belong to the contour points.

[0093] According to the assignment optimization result, a target block QP offset mask map may be determined. In some embodiments of the present disclosure, the generation module 402 may directly determine the assignment optimization result as the target block QP offset mask map, or the generation module 402 may determine the assignment optimization result as the intermediate block QP offset mask map, and may obtain the average value of the values ​​of each pixel point in the intermediate block QP offset mask map, and then may add the average value to the values ​​of each pixel point in the intermediate block QP offset mask map to obtain the target block QP offset mask map.

[0094] Furthermore, the allocation module 403 may allocate code rates to each image block in the to-be-processed image according to the obtained target block QP offset mask map.

[0095] In some embodiments of the present disclosure, the allocation module 403 may perform the following processing for any image block in the image to be processed: obtain the initial block QP of the image block determined according to the AQ algorithm, and superimpose the values ​​of the corresponding pixel points of the image block in the target block QP offset mask map on the initial block QP to obtain the bit rate allocation result of the image block.

[0096] Figure 4 The specific working process of the illustrated device embodiment can refer to the relevant description in the aforementioned method embodiment and will not be described in detail.

[0097] The solution disclosed in the present disclosure can be applied to the field of artificial intelligence, especially computer vision, image processing, video encoders and other fields. Artificial intelligence is a discipline that studies how to use computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It has both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing. Artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, machine learning / deep learning, big data processing technology, knowledge graph technology and other major directions.

[0098] The images and ROI area detection results in the embodiments of the present disclosure are not targeted at a specific user and do not reflect the personal information of a specific user. In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good customs.

[0099] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0100] Figure 5 A schematic block diagram of an electronic device 500 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0101] like Figure 5As shown, the device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 to a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0102] A number of components in the device 500 are connected to the I / O interface 505, including: an input unit 506, such as a keyboard, a mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0103] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI, Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSP, Digital Signal Processing), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 501 performs the various methods and processes described above, such as the methods described in the present disclosure. For example, in some embodiments, the methods described in the present disclosure may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the methods described in the present disclosure may be executed. Alternatively, in other embodiments, the computing unit 501 may be configured to execute the method described in the present disclosure in any other appropriate manner (for example, by means of firmware).

[0104] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard parts (ASSPs), system on chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0105] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0106] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory (EPROM, Electronically Programmable Read-Only Memory), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM, Compact Disc Read-Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0107] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0108] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0109] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0110] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0111] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A code rate allocation method, comprising: Obtaining N types of interest region detection results of the image to be processed, where N is a positive integer greater than one, and the N types of interest region detection results include: detection results obtained by performing interest region detection on the image to be processed using N different detection operators respectively; Generating a target block quantization parameter offset mask map corresponding to the image to be processed according to the N types of region of interest detection results; The code rate is allocated to each image block in the image to be processed according to the target block quantization parameter offset mask map.

2. The method according to claim 1, wherein: The step of generating a target block quantization parameter offset mask map corresponding to the image to be processed includes: Generating an initial block quantization parameter offset mask map according to the image to be processed; According to the N types of interest region detection results and the quantization parameter offset values ​​respectively corresponding to the N types of interest region detection results, the initial block quantization parameter offset mask map is assigned and optimized, and the target block quantization parameter offset mask map is determined according to the assignment optimization result.

3. The method according to claim 2, wherein: The initial block quantization parameter offset mask image is of size M*N, where the value of M is equal to the ratio of the length of the image to be processed to L, and the value of N is equal to the ratio of the width of the image to be processed to L; Furthermore, the value of each pixel in the initial block quantization parameter offset mask image is 0, and each pixel corresponds to an image block of size L*L in the image to be processed, and there is no overlap between any two image blocks.

4. The method according to claim 3, wherein: The optimizing the initial block quantization parameter offset mask map includes: The N types of interest region detection results are traversed in order from low to high according to the preset priority, and the following processing is performed for each traversed interest region detection result: The currently traversed detection result of the region of interest is used as the detection result to be processed, the pixel points matching the detection result to be processed are determined from the initial block quantization parameter offset mask map, and the values ​​of the matching pixel points are set as the quantization parameter offset values ​​corresponding to the detection result to be processed.

5. The method according to claim 4, wherein: In response to determining that the detection result to be processed is a detection result in the form of a rectangular frame, determining, from the initial block quantization parameter offset mask map, pixel points matching the detection result to be processed includes: The pixel points in the initial block quantization parameter offset mask image that meet the following requirements are determined as the matching pixel points: the image block corresponding to the pixel point overlaps with the rectangular frame.

6. The method according to claim 4, wherein: In response to determining that the detection result to be processed is a salient area mask map, determining the pixel points matching the detection result to be processed from the initial block quantization parameter offset mask map includes: Pixel points in the initial block quantization parameter offset mask map that meet the following requirements are determined as the matching pixel points: the proportion of the number of first-category pixel points in the image block corresponding to the pixel point in the total number of pixel points in the image block corresponding to the pixel point is greater than a first threshold, the first-category pixel points are pixel points whose value is 1 in the salient area mask map, the size of the salient area mask map is the same as that of the image to be processed, and the value of each pixel point in the salient area mask map is 1 or 0 respectively.

7. The method according to claim 4, wherein: In response to determining that the detection result to be processed is a contour point, determining a pixel point matching the detection result to be processed from the initial block quantization parameter offset mask map includes: The pixel points in the initial block quantization parameter offset mask image that meet the following requirements are determined as the matching pixel points: the proportion of the second type of pixel points in the image block corresponding to the pixel point in the total number of pixel points in the image block corresponding to the pixel point is greater than a second threshold, and the second type of pixel points are pixel points belonging to the contour point.

8. The method according to any one of claims 3 to 7, wherein: Determining the target block quantization parameter offset mask map according to the assignment optimization result includes: Determining the assignment optimization result as the target block quantization parameter offset mask map; Alternatively, the assignment optimization result is determined as an intermediate block quantization parameter offset mask map, the mean value of each pixel point in the intermediate block quantization parameter offset mask map is obtained, and the value of each pixel point in the intermediate block quantization parameter offset mask map is added to the mean value to obtain the target block quantization parameter offset mask map.

9. The method according to any one of claims 3 to 7, wherein: The allocating bit rates to each image block in the image to be processed comprises: For any image block in the image to be processed, the following processing is performed respectively: an initial block quantization parameter of the image block determined according to an adaptive quantization algorithm is obtained, and the values ​​of the corresponding pixel points of the image block in the target block quantization parameter offset mask map are superimposed on the initial block quantization parameter to obtain a bit rate allocation result of the image block.

10. A code rate allocation device, comprising: Acquisition module, generation module and allocation module; The acquisition module is used to acquire N types of interest region detection results of the image to be processed, where N is a positive integer greater than one, and the N types of interest region detection results include: detection results obtained by performing interest region detection on the image to be processed using N different detection operators respectively; The generating module is used to generate a target block quantization parameter offset mask map corresponding to the image to be processed according to the N types of region of interest detection results; The allocation module is used to allocate code rates to each image block in the image to be processed according to the target block quantization parameter offset mask map.

11. The device according to claim 10, wherein: The generation module generates an initial block quantization parameter offset mask map according to the image to be processed, and optimizes the initial block quantization parameter offset mask map according to the N types of region of interest detection results and the quantization parameter offset values ​​corresponding to the N types of region of interest detection results, and determines the target block quantization parameter offset mask map according to the assignment optimization result.

12. The device according to claim 11, wherein The initial block quantization parameter offset mask image is of size M*N, where the value of M is equal to the ratio of the length of the image to be processed to L, and the value of N is equal to the ratio of the width of the image to be processed to L; Furthermore, the value of each pixel in the initial block quantization parameter offset mask image is 0, and each pixel corresponds to an image block of size L*L in the image to be processed, and there is no overlap between any two image blocks.

13. The device according to claim 12, wherein: The generation module traverses the N types of interest region detection results in order from low to high according to a preset priority, and performs the following processing on each traversed interest region detection result: taking the currently traversed interest region detection result as the detection result to be processed, determining the pixel points that match the detection result to be processed from the initial block quantization parameter offset mask map, and setting the value of the matching pixel points to the quantization parameter offset value corresponding to the detection result to be processed.

14. The device according to claim 13, wherein: In response to determining that the detection result to be processed is a detection result in the form of a rectangular frame, the generation module determines the pixel points in the initial block quantization parameter offset mask image that meet the following requirements as the matching pixel points: the image block corresponding to the pixel points overlaps with the rectangular frame.

15. The device according to claim 13, wherein: In response to determining that the detection result to be processed is a salient area mask map, the generation module determines the pixel points in the initial block quantization parameter offset mask map that meet the following requirements as the matching pixel points: the proportion of the number of first-category pixel points in the image block corresponding to the pixel point in the total number of pixel points in the image block corresponding to the pixel point is greater than a first threshold, the first-category pixel points are pixel points whose value is 1 in the salient area mask map, the size of the salient area mask map is the same as that of the image to be processed, and the value of each pixel point in the salient area mask map is 1 or 0 respectively.

16. The device according to claim 13, wherein: In response to determining that the detection result to be processed is a contour point, the generation module determines the pixel points in the initial block quantization parameter offset mask image that meet the following requirements as the matching pixel points: the proportion of the second type of pixel points in the image block corresponding to the pixel point in the total number of pixel points in the image block corresponding to the pixel point is greater than a second threshold, and the second type of pixel points are pixel points belonging to the contour point.

17. The device according to any one of claims 12 to 16, wherein: The generation module determines the assignment optimization result as the target block quantization parameter offset mask map, or determines the assignment optimization result as the intermediate block quantization parameter offset mask map, obtains the mean value of each pixel point in the intermediate block quantization parameter offset mask map, and adds the mean value to the value of each pixel point in the intermediate block quantization parameter offset mask map to obtain the target block quantization parameter offset mask map.

18. The device according to any one of claims 12 to 16, wherein: The allocation module performs the following processing on any image block in the image to be processed: obtains the initial block quantization parameter of the image block determined according to the adaptive quantization algorithm, and superimposes the values ​​of the corresponding pixel points of the image block in the target block quantization parameter offset mask map onto the initial block quantization parameter to obtain the bit rate allocation result of the image block.

19. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.

20. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to make a computer execute the method according to any one of claims 1 to 9.

21. A computer program product, comprising a computer program / instruction, wherein when the computer program / instruction is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images