Image region of interest detection method, video encoding method, device, computer equipment, storage medium and computer program product

Abstract

The present application relates to an image region of interest detection method, video encoding method, apparatus, computer equipment, storage medium, and computer program product. The image region of interest detection method comprises: obtaining a current image frame to be detected; determining a mapping block size parameter value in encoding parameters corresponding to the current image frame to be detected, and based on the mapping block size parameter value, dividing the current image frame to be detected into multiple mapping blocks; performing texture region of interest detection on each mapping block to obtain a texture region of interest detection result; performing motion region of interest detection on each mapping block to obtain a motion region of interest detection result; and determining the region of interest corresponding to the current image frame to be detected based on the texture region of interest detection result and the motion region of interest detection result. This method can improve the accuracy of region of interest detection.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image region of interest detection method, video encoding method, apparatus, computer equipment, storage medium and computer program product. Background Art

[0002] With the advent of the video explosion era, demand for video quality is growing ever higher, and video bitrates are increasing exponentially. This poses significant challenges to operators in terms of network bandwidth costs and user viewing experience in the short term. Faced with these challenges, continuous optimization of video coding standards is essential to improve coding efficiency. In addition to the continuous introduction of new coding standards, AI technologies such as region of interest coding (ROI) are becoming increasingly important. Numerous subjective market reviews have shown that ROI coding significantly improves overall visual quality compared to traditional coding, especially in lower bandwidth environments. ROI coding achieves the desired high-quality image while maintaining a low bitrate, effectively resolving the trade-off between bitrate and image quality.

[0003] Region of interest (ROI)-based video coding utilizes the subjective existence of regions of interest (ROI) and regions of non-interest (NON-ROI) in the human eye, and makes certain corrections to the bit rate distribution between ROI and non-ROI, allocating more bit rate to the ROI, thereby subjectively improving video quality.

[0004] Currently, many hardware products focus solely on hardware resources and encoding performance. Many manufacturers' hardware video encoding devices do not implement ROI detection algorithms, but instead provide interfaces that support ROI encoding, allowing users to set ROIs and adjust the quantization parameter (QP) of already set ROIs. Most products, such as the x265 open source project (HEVC's open source encoding library), add ROI detection algorithms to support ROI encoding in upper-layer applications. This includes the x265 open source project (HEVC's open source encoding library). The x265 open source project implements similar ROI video encoding using adaptive quantization (AQ) technology. The AQ algorithm in x265 recursively calculates the mean and variance of 64x64, 32x32, 16x16, and 8x8 blocks to derive the quantization parameter offset (QP-delta) for these four block types across the entire encoded frame. This implementation places high demands on hardware in terms of computational complexity and storage. Currently, the engineering community is making minor adjustments to the existing x265 AQ algorithm to achieve ROI video encoding support at the software application level.

[0005] In addition, research on region of interest coding includes perceptual coding based on the human visual system (HVS). The HVS has many parameters and a complex model, and the model construction is inaccurate, which leads to inaccurate detection of the region of interest in the video frame. Summary of the Invention

[0006] Based on this, it is necessary to provide an image region of interest detection method, video encoding method, device, computer equipment, computer-readable storage medium and computer program product that can improve the accuracy of region of interest detection in response to the above technical problems.

[0007] In a first aspect, the present application provides a method for detecting a region of interest in an image, the method comprising:

[0008] Get the current image frame to be detected;

[0009] Determining a mapping block size parameter value in the encoding parameters corresponding to the current image frame to be detected, and dividing the current image frame to be detected into a plurality of mapping blocks based on the mapping block size parameter value;

[0010] Performing texture region of interest detection on each of the mapping blocks to obtain a texture region of interest detection result;

[0011] Performing motion region of interest detection on each of the mapping blocks to obtain a motion region of interest detection result;

[0012] Based on the texture region of interest detection result and the motion region of interest detection result, a region of interest corresponding to the current image frame to be detected is determined.

[0013] In one embodiment, performing texture region of interest detection on each of the mapping blocks to obtain a texture region of interest detection result includes:

[0014] Dividing each of the mapping blocks into sub-blocks, and determining the size of the sub-blocks;

[0015] Obtaining a first threshold adjustment factor corresponding to the current image frame to be detected;

[0016] Obtaining a quantization parameter value of the current image frame to be detected;

[0017] Determining a first initialization threshold based on the quantization parameter value of the current image frame to be detected and the size of the sub-block;

[0018] Performing region of interest detection on the sub-block based on the first initialization threshold and the first threshold adjustment factor to obtain a texture region of interest detection result of the mapping block.

[0019] In one embodiment, determining the first initialization threshold based on the quantization parameter value of the current image frame to be detected and the size of the sub-block includes:

[0020] Querying a first mapping relationship table to determine a first initialization threshold corresponding to the quantization parameter value and the size of the sub-block;

[0021] When the first initialization threshold corresponding to the quantization parameter value and the size of the sub-block is not found in the first mapping relationship table, the first initialization threshold corresponding to the quantization parameter value and the size of the sub-block is determined based on the quantization parameter value and the size of the sub-block in the first mapping relationship table by interval size positioning.

[0022] In one embodiment, the performing ROI detection on the sub-block based on the first initialization threshold and the first threshold adjustment factor to obtain a texture ROI detection result of the mapping block includes:

[0023] Obtaining an edge pixel determination threshold based on a first threshold adjustment factor and the first initialization threshold;

[0024] Calculating an edge gradient value of a pixel in each of the sub-blocks; and determining an edge pixel determination result of the pixel based on the edge gradient value of the pixel and the edge pixel determination threshold;

[0025] Determining an edge density value of the corresponding sub-block based on each edge pixel judgment result, and determining that the sub-block is a region of interest when the edge density value is greater than an edge density threshold;

[0026] When the mapping block includes at least one sub-block that is a region of interest, the mapping block is determined to be a region of interest.

[0027] In one embodiment, obtaining the first threshold adjustment factor corresponding to the current image frame to be detected includes:

[0028] When the current image frame to be detected is an initial frame, obtaining a first initialization adjustment factor as a first threshold adjustment factor;

[0029] When the current image frame to be detected is not an initial frame, a first threshold adjustment factor is obtained based on the number of mapping blocks of the region of interest and the number of mapping blocks of the region of no interest in the texture region detection result of the previous image frame to be detected.

[0030] In one embodiment, performing motion region of interest detection on each mapping block to obtain a motion region of interest detection result of the mapping block includes:

[0031] Obtaining a quantization parameter value of the current image frame to be detected;

[0032] Obtaining a second threshold adjustment factor corresponding to the current image frame to be detected;

[0033] determining second initialization thresholds for different motion vector directions based on the quantization parameter value;

[0034] performing motion detection on the prediction units corresponding to the mapping blocks based on the second initialization threshold and the second threshold adjustment factor, and calculating the total area of ​​the motion region in each mapping block based on the motion detection results of the prediction units;

[0035] When the ratio of the total area of ​​the motion regions in the mapping block to the area of ​​the mapping block is greater than a proportion threshold, the mapping block is determined to be a region of interest.

[0036] In one embodiment, determining the second initialization thresholds for different motion vector directions based on the quantization parameter value includes:

[0037] Querying a second mapping relationship table to determine second initialization thresholds for different motion vector directions corresponding to the quantization parameter values;

[0038] When the second initialization thresholds for different motion vector directions corresponding to the quantization parameter values ​​are not found in the second mapping relationship table, the second initialization thresholds for different motion vector directions corresponding to the quantization parameter values ​​are determined based on the quantization parameter values ​​in the second mapping table by means of interval size positioning.

[0039] In one embodiment, the performing motion detection on the prediction unit corresponding to each mapping block based on the second initialization threshold and the second threshold adjustment factor, and calculating the total area of ​​the motion region in each mapping block based on the motion detection result of each prediction unit, includes:

[0040] Determining a motion vector threshold corresponding to each different motion vector direction based on the second initialization threshold for each different motion vector direction and the second threshold adjustment factor;

[0041] Obtaining a mapping block at a corresponding position of a previous image frame to be detected relative to the current image frame to be detected;

[0042] When the motion vectors of the prediction unit included in the acquired mapping block in different directions are all greater than the corresponding motion vector threshold, determining that the prediction unit is a motion area;

[0043] Based on the areas of the prediction units determined to be motion areas, the total area of ​​the motion areas in the mapping block is obtained.

[0044] In one embodiment, obtaining the second threshold adjustment factor corresponding to the current image frame to be detected includes:

[0045] When the image frame to be detected is a target initial frame, obtaining a second initialization adjustment factor as a second threshold adjustment factor;

[0046] When the image frame to be detected is not the target initial frame, a second threshold adjustment factor is obtained based on the ratio of the area of ​​the motion area in the target image frame to be detected to the area of ​​the motion area in the previous image frame to be detected, wherein the target image frame to be detected is the previous frame of the previous image frame to be detected.

[0047] In one embodiment, determining the region of interest corresponding to the current image frame to be detected based on the texture region of interest detection result and the motion region of interest detection result includes:

[0048] determining an interest type and an interest intensity of each mapping block based on the texture ROI detection result and the motion ROI detection result of the mapping block;

[0049] The region of interest corresponding to the current image frame to be detected is obtained based on the interest type and interest intensity of each mapping block.

[0050] In one embodiment, the method further comprises:

[0051] Outputting the region of interest corresponding to the current image frame to be detected to a quantization parameter value calculation module of the hardware; or

[0052] The region of interest corresponding to the current image frame to be detected is output to a target storage area, and the region of interest corresponding to the current image frame to be detected in the target storage area is transmitted to a hardware encoder through a user interface.

[0053] In a second aspect, the present application further provides a video encoding method, the video encoding method comprising:

[0054] Obtaining a region of interest obtained by the method for detecting a region of interest in an image according to any one of the above embodiments;

[0055] Based on the region of interest, a quantization parameter value of the current image frame is calculated;

[0056] The image frame is encoded based on the quantization parameter value.

[0057] In a third aspect, the present application further provides an image region of interest detection device, the device comprising:

[0058] The image acquisition module to be detected is used to obtain the current image frame to be detected;

[0059] a mapping block division module, configured to determine a mapping block size parameter value in the encoding parameters corresponding to the current image frame to be detected; and based on the mapping block size parameter value, divide the current image frame to be detected into a plurality of mapping blocks;

[0060] A texture region of interest detection module, configured to perform texture region of interest detection on each of the mapping blocks to obtain a texture region of interest detection result;

[0061] A motion region of interest detection module, configured to perform motion region of interest detection on each of the mapping blocks to obtain a motion region of interest detection result;

[0062] The region of interest determination module is configured to determine the region of interest corresponding to the current image frame to be detected based on the texture region of interest detection result and the motion region of interest detection result.

[0063] In a fourth aspect, the present application further provides a video device, comprising:

[0064] A region of interest acquisition module, used to acquire the region of interest obtained based on the above-mentioned image region of interest detection device;

[0065] A quantization parameter value updating module, configured to calculate a quantization parameter value of a current image frame based on the region of interest;

[0066] The encoding module is configured to perform image frame encoding based on the quantization parameter value.

[0067] In a fifth aspect, the present application also provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method in any one of the above embodiments when executing the computer program.

[0068] In a sixth aspect, the present application also provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the method in any one of the above-mentioned embodiments when executed by a processor.

[0069] In a seventh aspect, the present application also provides a computer program product, comprising a computer program, which implements the steps of the method in any one of the above embodiments when executed by a processor.

[0070] The above-mentioned image region of interest detection method, video encoding method, device, computer equipment, storage medium and computer program product, after obtaining the current image frame to be detected, obtain the mapping block size parameter from its corresponding encoding parameter, and based on the mapping block size parameter value, divide the current image frame to be detected into multiple mapping blocks, and then perform texture region of interest detection on each mapping block to obtain a texture region of interest detection result; perform motion region of interest detection on each of the mapping blocks to obtain a motion region of interest detection result, and finally combine the texture region of interest detection result and the motion region of interest detection result to determine the region of interest corresponding to the current image frame to be detected, taking into account both the motion properties of the image frame and the static texture properties of the image frame, and can accurately detect the region of interest of the image frame to be detected. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0072] Figure 1 A diagram showing an application environment of a method for detecting an image region of interest in one embodiment;

[0073] Figure 2 1 is a flow chart of a method for detecting an image region of interest in one embodiment;

[0074] Figure 3 A flowchart of a texture region of interest detection step in one embodiment;

[0075] Figure 4 Schematic diagram of the relative positions of pixels in the gradient calculation of the Sobel operator in one embodiment;

[0076] Figure 5 is a flow chart of a step of detecting a moving region of interest in one embodiment;

[0077] Figure 6 1 is a flow chart of a video encoding method according to an embodiment;

[0078] Figure 7 is a structural block diagram of an image region of interest detection device in one embodiment;

[0079] Figure 8 is a structural block diagram of a video encoding device in one embodiment;

[0080] Figure 9FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0081] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0082] The image region of interest detection method provided in the embodiment of the present application can be applied to Figure 1 In the encoder shown, the encoder includes an encoder preprocessing module and an encoder module. The encoder preprocessing module mainly calculates encoding parameters for the encoder module to assist the encoder in encoding. The encoder performs the normal encoding process of the image frame: inputting the current image frame and the image frame that has been encoded, then performing inter-frame / intra-frame prediction, transformation, quantization, entropy coding, and finally outputting the encoded bitstream.

[0083] The image region of interest detection method of the present application is implemented in the encoder preprocessing module, which can be executed by corresponding hardware, and the execution of this method can be implemented in parallel with the formal hardware encoder. In this way, the image region of interest detection method is implemented in the encoder preprocessing module, which only serves as a pre-analysis stage of the formal encoder. The encoder preprocessing module and the formal encoder are two parallel encoding processes, and the encoder preprocessing module does not affect the encoding performance of the formal encoder.

[0084] Combine Figure 1 As shown, the hardware module corresponding to the image region of interest detection method includes two parts of input: the first part of the input is the original image data of the current image frame to be detected, and the second part of the input is the motion vector generated by the current image frame to be detected during the inter-frame prediction process. In addition, the image region of interest detection algorithm also needs to obtain the size of the mapping block specified in the normal encoding parameters of the encoder. The mapping block is the smallest calculation unit in the region of interest detection method, that is, by dividing the current image frame to be detected into multiple mapping blocks, and then judging whether each mapping block is a region of interest, the region of interest of the current image frame to be detected is obtained. Among them, the image region of interest detection method uses the mapping block size (mapblockSize) as an input parameter, supports different block sizes from 128, 64, 32, and 16, and complies with and is compatible with the current mainstream video coding standard based on variable block encoding.

[0085] Among them, detecting whether each mapping block in the current image frame to be detected is a region of interest takes into account both the motion properties of the current image frame to be detected and the static texture properties of the current image frame to be detected, and can accurately detect the region of interest of the current image frame to be detected; at the same time, the first initialization threshold and the second initialization threshold in the method are obtained by looking up the quantization parameter (Qp) of the coding frame input, so that different quantization parameters (Qp) index different first initialization thresholds and second initialization thresholds, and the first initialization threshold and the second initialization threshold can be adaptively adjusted according to the coding information of the current image frame to be detected. This adaptive performance ensures the correctness of the detection of the region of interest of the current image frame to be detected.

[0086] Texture region of interest detection for mapping blocks can be performed by using the Sobel gradient operator combined with threshold judgment, and motion region of interest detection for mapping blocks can be performed by using the motion vector combined with threshold judgment. In this way, the image processing algorithm based on the Sobel operator only involves the multiplication and addition operation of the input video frame pixel value and the Sobel operator, and the implementation is relatively simple; the motion region of the video frame is detected based on the motion vector (MV) information of the encoded frame, which can be achieved by only caching these data, and the algorithm implementation complexity is low.

[0087] Continue to combine Figure 1 As shown, the final output data of the region of interest detection method is the region of interest of the current image frame to be detected. Optionally, the data output form that can be adopted is: Indicates the coordinates of the upper left pixel of the region of interest, Indicates the width of the rectangular area of ​​interest, Represents the height of the rectangular area of ​​interest, The interest intensity of the region of interest is used to guide and constrain the calculation of the quantization parameter (Qp) offset value (Qp-delta).

[0088] In one optional embodiment, the output of the region of interest may include at least one of the following: direct output to a quantization parameter calculation module of a hardware encoder; output to a target storage area, such as writing to an external memory, so that it is passed as input to other hardware encoders that support region of interest encoding through a user interface.

[0089] In one embodiment, an encoder module and an encoder preprocessing module are executed in parallel, wherein the encoder module is used to encode an image. Specifically, the encoding method includes: obtaining a region of interest (ROI) obtained based on an image region of interest detection method; calculating a quantization parameter value for a current image frame based on the ROI; and encoding the image frame based on the quantization parameter value.

[0090] The encoder can be applied to any terminal or server to encode video. Terminals include, but are not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices include smart speakers, smart TVs, smart air conditioners, and smart car devices. Portable wearable devices include smart watches, smart bracelets, and head-mounted devices. The server can be implemented as a standalone server or a server cluster consisting of multiple servers.

[0091] In an exemplary embodiment, Figure 2 As shown, a method for detecting regions of interest in an image is provided. Figure 1 The encoder preprocessing module in FIG. 1 is taken as an example to illustrate the method, which includes the following steps S202 to S210. In which:

[0092] S202: Acquire the current image frame to be detected.

[0093] The current image frame to be detected is the current frame in the video, and can be regarded as the next frame to be encoded.

[0094] S204: Determine a mapping block size parameter value in the encoding parameters corresponding to the current image frame to be detected, and divide the current image frame to be detected into a plurality of mapping blocks based on the mapping block size parameter value.

[0095] The mapping block size parameter value is stored in the parameters normally encoded by the encoder, and the mapping block is the smallest calculation unit of the region of interest detection method. For example, if the region of interest encoding is used in the encoding process, the size of the mapping block is set by the encoding parameters, so that the mapping block size parameter value can be directly obtained from the encoding parameters. In some optional embodiments, the mapping block size parameter value may include but is not limited to 128 pixels. 128 pixels, 64 pixels 64 pixels, 32 pixels 32 pixels and 16 pixels 16 pixels.

[0096] The encoder preprocessing module divides the current image frame to be detected into multiple mapping blocks based on the mapping block size parameter value, wherein the order of division can be starting from the upper left corner pixel of the current image frame to be detected.

[0097] S206: Perform texture region of interest detection on each mapping block to obtain a texture region of interest detection result.

[0098] S208: Perform motion region of interest detection on each mapping block to obtain a motion region of interest detection result.

[0099] In the time domain, moving areas of a video image are particularly attractive; in the spatial domain, edge information and areas with complex textures in a video image are particularly attractive. To this end, this application performs texture ROI detection and motion ROI detection on each mapping block to determine whether each mapping block is a ROI. The texture ROI detection result is the result of determining whether the mapping block is a ROI based on the texture. The motion ROI detection result is the result of determining whether the mapping block is a ROI based on the motion detection.

[0100] The texture region of interest detection for the mapping block may be performed by using a Sobel gradient operator in combination with a threshold value judgment. In other embodiments, other texture region of interest detection operators may also be used.

[0101] The motion ROI detection for the mapping block may be performed by using a motion vector combined with a threshold value judgment. In other embodiments, other motion detection methods may also be used.

[0102] Among them, the initial values ​​of the thresholds involved in texture region of interest detection and motion region of interest detection can be obtained by looking up the table according to the quantization parameters of the encoder, and adaptively adjusted during the encoding process to ensure the accuracy of the texture region of interest detection results and the motion region of interest detection results.

[0103] S210: Determine a region of interest corresponding to the current image frame to be detected based on the texture region of interest detection result and the motion region of interest detection result.

[0104] The encoder preprocessing module obtains the result of whether each mapping block is a region of interest based on the texture region of interest detection results and the motion region of interest detection results, and then obtains the region of interest of the current image frame to be detected based on all the mapping blocks determined to be regions of interest.

[0105] In one of the optional embodiments, based on the texture region of interest detection results and the motion region of interest detection results, the region of interest corresponding to the current image frame to be detected is determined, including: based on the texture region of interest detection results and the motion region of interest detection results of the mapping blocks, determining the interest type and interest intensity of each mapping block; based on the interest type and interest intensity of each mapping block, obtaining the region of interest corresponding to the current image frame to be detected.

[0106] The types of interest may be predefined. In one optional embodiment, the types of interest are divided into four different types based on different detection methods. In other embodiments, the types of interest may be divided into other numbers of types. No specific limitation is made here. In this embodiment, the types of interest are divided into four different types based on the texture region of interest detection method and the motion region of interest detection method. For details, please refer to Table 1.

[0107] The interest intensity is set for different interest types. Different interest types correspond to different interest intensities. The value of the interest intensity may be preset, and those skilled in the art may set different values ​​based on requirements.

[0108] Table 1

[0109]

[0110] Based on Table 1, when both the texture ROI detection result and the motion ROI detection result indicate that the mapping block is a ROI, the ROI type of the mapping block is 0 and the interest intensity is 6; when both the texture ROI detection result and the motion ROI detection result indicate that the mapping block is not a ROI, the ROI type of the mapping block is 3 and the interest intensity is -2; when the texture ROI detection result indicates that the mapping block is a ROI and the motion ROI detection result indicates that the mapping block is not a ROI, the ROI type of the mapping block is 1 and the interest intensity is 3. When the texture ROI detection result indicates that the mapping block is not a ROI and the motion ROI detection result indicates that the mapping block is a ROI, the ROI type of the mapping block is 2 and the interest intensity is 2.

[0111] Based on Table 1, the output data of the region of interest detection algorithm .

[0112] In one of the optional embodiments, the region of interest detection algorithm outputs data The output method includes at least one of the following: outputting the region of interest corresponding to the current image frame to be detected to the hardware quantization parameter value calculation module; or outputting the region of interest corresponding to the current image frame to be detected to the target storage area, and the region of interest corresponding to the current image frame to be detected in the target storage area is transmitted to the hardware encoder through the user interface.

[0113] When the encoder has its own quantization parameter value calculation module, the encoder preprocessing module outputs the detection result of the region of interest to the hardware quantization parameter value calculation module to generate the quantization parameter offset value required by the encoder module.

[0114] Another method is that the encoder preprocessing module outputs the detection result of the region of interest to a target storage area, such as an external memory, and transmits it as input to other hardware encoders that support region of interest encoding through a user interface.

[0115] The above-mentioned image region of interest detection method obtains the mapping block size parameter from the corresponding encoding parameter after obtaining the current image frame to be detected, and divides the current image frame to be detected into multiple mapping blocks based on the mapping block size parameter value, and then performs texture region of interest detection on each mapping block to obtain a texture region of interest detection result; performs motion region of interest detection on each mapping block to obtain a motion region of interest detection result, and finally combines the texture region of interest detection result and the motion region of interest detection result to determine the region of interest corresponding to the current image frame to be detected, which takes into account both the motion properties of the image frame and the static texture properties of the image frame, and can accurately detect the region of interest of the image frame to be detected.

[0116] In one of the optional embodiments, combined with Figure 3 As shown, Figure 3 Flowchart of a texture region of interest detection step in one embodiment. In this embodiment, the texture region of interest detection step, i.e., performing texture region of interest detection on each mapping block to obtain a texture region of interest detection result, includes:

[0117] S302: Divide each mapping block into sub-blocks, and determine the size of the sub-blocks.

[0118] Sub-blocks are obtained by dividing a mapping block. Optionally, each mapping block is divided into A sub-blocks. In one optional embodiment, A = 4. In other embodiments, A can be other values. However, if the value of A is greater than 4, the hardware implementation cost is relatively high. Dividing a mapping block into multiple sub-blocks allows for detecting the complexity of block textures at a smaller block granularity, thereby locating the size of the complex texture region of the entire mapping block.

[0119] After the mapping block is divided, the size of the sub-block is obtained based on the size of the mapping block and the number of sub-blocks. For example, the size of the mapping block is 128 pixels. 128 pixels, 64 pixels 64 pixels, 32 pixels 32 pixels and 16 pixels 16 pixels, the corresponding sub-block size is 64 pixels 64 pixels, 32 pixels 32 pixels, 16 pixels 16 pixels and 8 pixels 8 pixels.

[0120] S304: Obtain a first threshold adjustment factor corresponding to the current image frame to be detected.

[0121] The first threshold adjustment factor is a factor for adjusting the first initialization threshold, and the initial value of the first threshold adjustment factor is 1. Subsequently, based on the number of mapping blocks of the texture ROI and the number of mapping blocks of the non-texture ROI in the current image frame to be detected, the first threshold adjustment factor of the next image frame to be detected is calculated and stored for use when performing texture ROI detection on the current image frame to be detected.

[0122] In one of the optional embodiments, a first threshold adjustment factor corresponding to the current image frame to be detected is obtained, including: if the current image frame to be detected is an initial frame, obtaining a first initialization adjustment factor as a first threshold adjustment factor; if the current image frame to be detected is not an initial frame, obtaining the first threshold adjustment factor based on the number of mapping blocks of the area of ​​interest and the number of mapping blocks of the area of ​​non-interest in the texture area of ​​interest detection result of the previous frame of the image frame to be detected.

[0123] Optionally, that is, when the current image frame to be detected is an initial frame, that is, the first frame to be encoded, the first threshold adjustment factor is the first initialization adjustment factor, that is, 1. If the current image frame to be detected is not the first frame, the generated first threshold adjustment factor is obtained. Specifically, the first threshold adjustment factor is calculated as follows:

[0124] (1)

[0125] Wherein, the video coding sequence includes multiple coding frames, i.e., image frames to be detected, n represents the index of the coding frame in the coding sequence, Indicates the number of mapping blocks detected as regions of interest in coded frame n. Indicates the number of non-region-of-interest mapping blocks in coded frame n.

[0126] Among them, the first threshold adjustment factor is calculated After that, it is stored to generate a threshold for edge pixel judgment of the next coding frame.

[0127] S306: Obtain the quantization parameter value of the current image frame to be detected.

[0128] The quantization parameter value is obtained from the image parameter set of the current image frame to be detected, which is used for the encoding of interest.

[0129] S308: Determine a first initialization threshold based on the quantization parameter value of the current image frame to be detected and the size of the sub-block.

[0130] The first initialization threshold is used for edge pixel judgment, and can be obtained by inputting a quantization parameter value and a sub-block size, for example, by looking up a table based on the quantization parameter value and the sub-block size.

[0131] In one of the optional embodiments, a first initialization threshold is determined based on the quantization parameter value and the sub-block size of the current image frame to be detected, including: querying a first mapping relationship table to determine the first initialization threshold corresponding to the quantization parameter value and the sub-block size; when the first initialization threshold corresponding to the quantization parameter value and the sub-block size is not found in the first mapping relationship table, based on the quantization parameter value and the sub-block size in the first mapping relationship table, the first initialization threshold corresponding to the quantization parameter value and the sub-block size is determined by interval size positioning.

[0132] In this embodiment, the first initialization threshold used for texture region of interest detection is obtained by looking up the table according to the quantization parameter QP of the video frame and the sub-block size N. Common values ​​of the quantization parameter Qp include 24, 28, 32, and 36, and other values ​​may be used in other embodiments; the sub-block size N is the size of the divided sub-blocks, and common values ​​thereof include 8, 16, 32, and 64, and other values ​​may be used in other embodiments.

[0133] The combination of Table 2 to Table 5 shows the quantization parameter QP, sub-block size N and the first initialization threshold. The table is divided into several sub-tables, where Table 2 shows the quantization parameter QP and the first initialization threshold when N=8. Table 3 shows the relationship between the quantization parameter QP and the first initialization threshold when N=16. Table 4 shows the relationship between the quantization parameter QP and the first initialization threshold when N=32. Table 5 shows the relationship between the quantization parameter QP and the first initialization threshold when N=64. Relationship table.

[0134] Table 2 N=8

[0135]

[0136] Table 3 N=16

[0137]

[0138] Table 4 N=32

[0139]

[0140] Table 5 N=64

[0141]

[0142] The threshold table for boundary judgment was developed by adding Sobel edge detection code to the official HM of the HEVC (high-efficiency video coding) standard after extensive coding tests. Table 2-5 lists only the threshold judgment values ​​for commonly used quantization parameters (Qp). For thresholds of quantization parameters not listed in the table, the corresponding values ​​can be obtained based on the interval size positioning method. For example, if the input quantization parameter Qp=22 is smaller than Qp=24 in the table, the threshold corresponding to Qp=24 can be used.

[0143] S310: Performing region of interest detection on the sub-block based on the first initialization threshold and the first threshold adjustment factor to obtain a texture region of interest detection result of the mapping block.

[0144] In one of the optional embodiments, a region of interest detection is performed on a sub-block based on a first initialization threshold and a first threshold adjustment factor to obtain a texture region of interest detection result of the mapping block, including: obtaining an edge pixel judgment threshold based on the first threshold adjustment factor and the first initialization threshold; calculating an edge gradient value of a pixel in each sub-block; determining an edge pixel judgment result of the pixel based on the edge gradient value of the pixel and the edge pixel judgment threshold; determining an edge density value of the corresponding sub-block based on each edge pixel judgment result, and when the edge density value is greater than the edge density threshold, determining the sub-block to be a region of interest; and determining the mapping block to be a region of interest when the mapping block includes at least one sub-block that is a region of interest.

[0145] The detection of the region of interest may include a step of calculating a pixel gradient value, a step of judging edge pixels based on the pixel gradient value, a first initialization threshold value, and a first threshold adjustment factor, a step of obtaining edge density based on the judgment result of the edge pixels, and a step of detecting a texture region of interest based on the edge density.

[0146] The calculation step of pixel gradient value is mainly to calculate sub-block N The edge gradient of pixel (i, j) in N , where it is assumed that the pixel is the pixel value at the position of NxN sub-block (i, j), then the calculation formula of the gradient value in the horizontal and vertical directions in Sobel edge detection is as follows:

[0147] (2)

[0148] (3)

[0149] in, , and Represents the gradient values ​​of pixel (i, j) in the horizontal and vertical directions, and the relative positions of the pixels involved in the calculation of the Sobel gradient value. 、 、 、 、 、 、 、 etc. can be combined Figure 4 As shown, it is the pixel value of the adjacent pixels of pixel (i, j), Figure 4 Schematic diagram of the relative positions of pixels in the gradient calculation of the Sobel operator in one embodiment. The gradient value of a position can be defined by the following formula:

[0150] (4)

[0151] The available values ​​of N are 8, 16, 32, and 64;

[0152] The step of edge pixel judgment based on pixel gradient value, first initialization threshold value and first threshold adjustment factor is to perform threshold judgment on the product of the calculated pixel gradient value, first initialization threshold value and first threshold adjustment factor, i.e., to judge and , calculate the edge density values ​​of the four NxN sub-blocks in the current detection map block :

[0153] (5)

[0154]

[0155] in, It is the first initialization threshold value obtained by looking up the table according to the quantization parameter (Qp) and the input sub-block size N; is the first threshold adjustment factor, the initial value .

[0156] The step of obtaining edge density based on the judgment result of edge pixels is to The edge density value of the sub-block Perform threshold judgment and calculate Whether a sub-block is a region of interest can be determined by the following formula:

[0157] (6)

[0158] represents a sub-block, yes The edge density threshold of the sub-block is obtained through experience. hour, The detection effect of the sub-blocks is in line with expectations. This value is obtained by adding the Sobel detection algorithm to the HM soft encoder and running different encoding instances as mentioned above. It generally does not need to be changed for user use. Unless the usage scenario is very demanding, users can analyze their own encoding instances to obtain appropriate values.

[0159] The steps of detecting the texture region of interest based on edge density include: if one of the NxN sub-blocks is the region of interest, the currently detected mapping block can be defined as the region of interest:

[0160] (7)

[0161] In the above embodiment, the first initialization threshold is obtained by looking up the table of the quantization parameter (Qp) of the current image frame to be detected, so that different quantization parameters (Qp) index different first initialization thresholds, and the first initialization threshold can be adaptively adjusted according to the encoding information of the current image frame to be detected. This adaptive performance ensures the correctness of the detection of the region of interest of the current image frame to be detected, and the image processing algorithm based on the Sobel operator only involves the multiplication and addition operation of the pixel value of the input current image frame to be detected and the Sobel operator, which is relatively simple to implement.

[0162] In one of the optional embodiments, combined with Figure 5 As shown, Figure 5 Flowchart of a motion region of interest detection step in one embodiment. In this embodiment, the motion region of interest detection step, i.e., performing motion region of interest detection on each mapping block to obtain a motion region of interest detection result of the mapping block, includes:

[0163] S502: Obtain the quantization parameter value of the current image frame to be detected.

[0164] The quantization parameter value is obtained from the image parameter set of the current image frame to be detected, which is used for the encoding of interest.

[0165] The texture region of interest detection step and the motion region of interest detection step both involve obtaining quantization parameter values. For convenience, they can be uniformly obtained from the image parameter set of the current image frame to be detected into the memory and can be used directly later.

[0166] S504: Obtain a second threshold adjustment factor corresponding to the current image frame to be detected.

[0167] The second threshold adjustment factor is a factor for adjusting the second initialization threshold, and the initial value of the first threshold adjustment factor is 1. Subsequently, based on the number of mapping blocks of the motion ROI in the current image frame to be detected and the number of mapping blocks of the motion ROI in the previous image frame to be detected, the second threshold adjustment factor of the next image frame to be detected is calculated, and the first threshold adjustment factor is stored for use in texture ROI detection in the current image frame to be detected.

[0168] In one of the optional embodiments, a second threshold adjustment factor corresponding to the current image frame to be detected is obtained, including: when the image frame to be detected is the target initial frame, obtaining a second initialization adjustment factor as the second threshold adjustment factor; when the image frame to be detected is not the target initial frame, obtaining a second threshold adjustment factor based on the ratio of the area of ​​the motion area in the target image frame to be detected to the area of ​​the motion area in the previous image frame to be detected, wherein the target image frame to be detected is the frame before the previous image frame to be detected.

[0169] Specifically, that is, when the current image frame to be detected is the target initial frame, that is, the first frame, the second frame, and the third frame to be encoded, the second threshold adjustment factor is the second initialization adjustment factor, that is, 1. When the image frame to be detected is not the target initial frame, the ratio of the area of ​​the motion region in the target image frame to be detected to the area of ​​the motion region in the previous image frame to be detected is used as the second threshold adjustment factor, wherein the target image frame to be detected is the frame before the previous image frame to be detected. Specifically, the second threshold adjustment factor The calculation is as follows:

[0170] (8)

[0171] in, Indicates the area of ​​the motion region in the previous frame (frame n-1) encoded before the current image frame to be detected (frame n); Indicates the area of ​​the motion region in the first two frames (frame n-2) encoded before the current image frame (frame n) to be detected.

[0172] Among them, the second threshold adjustment factor is calculated After that, it is stored for subsequent calculation of motion areas.

[0173] S506: Determine second initialization thresholds for different motion vector directions based on the quantization parameter value.

[0174] The second initialization threshold is used for motion region judgment, and can be obtained by looking up a table using a motion vector and a quantization parameter.

[0175] In one of the optional embodiments, based on the quantization parameter value, the second initialization threshold value for different motion vector directions is determined, including: querying the second mapping relationship table to determine the second initialization threshold value for different motion vector directions corresponding to the quantization parameter value; when the second initialization threshold value for different motion vector directions corresponding to the quantization parameter value is not found in the second mapping relationship table, based on the quantization parameter value in the second mapping table, the second initialization threshold value for different motion vector directions corresponding to the quantization parameter value is determined by interval size positioning.

[0176] In this embodiment, the second initialization thresholds for different motion vector directions used for texture region of interest detection are obtained based on the quantization parameter QP of the video frame and the motion vector table lookup. and Commonly used values ​​of the quantization parameter Qp include 24, 28, 32, and 36, and other values ​​may also be used in other embodiments.

[0177] As shown in Table 6, the second initialization thresholds corresponding to the quantization parameter QP and the values ​​MV.x and MV.y of the two vector directions of the motion vector (MV) are as follows:

[0178] Table 6

[0179]

[0180] Among them, Table 6 only gives the threshold data of the commonly used coding quantization parameter Qp. For the quantization parameter Qp not in Table 6, the corresponding threshold data can be obtained by comparing the interval size. For example, the coding quantization parameter Qp=22, Qp=22 is smaller than Qp=24, so Qp=22 can take the threshold data corresponding to Qp=24.

[0181] S508: Perform motion detection on the prediction units corresponding to each mapping block based on the second initialization threshold and the second threshold adjustment factor, and calculate the total area of ​​the motion region in each mapping block based on the motion detection results of each prediction unit.

[0182] In one of the optional embodiments, motion detection is performed on the prediction units corresponding to each mapping block based on a second initialization threshold and a second threshold adjustment factor, and the total area of ​​the motion region in each mapping block is calculated based on the motion detection results of each prediction unit, including: determining the motion vector threshold corresponding to each different motion vector direction based on the second initialization threshold and the second threshold adjustment factor for different motion vector directions; obtaining a mapping block at a corresponding position of an image frame to be detected that is previous to the current image frame to be detected; when the motion vectors of the prediction units in different directions included in the obtained mapping block are all greater than the corresponding motion vector threshold, determining that the prediction unit is a motion region; and obtaining the total area of ​​the motion region in the mapping block based on the area of ​​each prediction unit determined to be a motion region.

[0183] The determination of the motion region in the mapping block includes determining whether the corresponding prediction unit in the mapping block is a motion region, wherein the determination of whether the prediction unit (pu) is a motion region is based on a comparison of the motion vectors (MV) of all prediction units (pu) contained in the co-located mapping block of the previous encoded image frame to be detected with a motion vector threshold. The motion vector threshold is obtained based on the second initialization threshold and the second threshold adjustment factor, that is, the product of the two. The determination of whether the prediction unit is a motion region can be obtained by the following formula:

[0184] (9)

[0185] MV.x and MV.y are the vector values ​​of the motion vector (MV) in the x and y directions of the current prediction unit (pu). and is the threshold for determining whether the prediction unit is a motion region, It is the adaptive adjustment factor of the motion vector threshold, and its initial value is 1.

[0186] In this way, based on the area of ​​each prediction unit determined to be a motion area, the total area of ​​the motion area in the mapping block is obtained, specifically:

[0187] (10)

[0188] in, is the total area of ​​the motion region in the mapping block, is the area of ​​each prediction unit determined to be a motion region.

[0189] S510: When the ratio of the total area of ​​the motion region in the mapping block to the area of ​​the mapping block is greater than a ratio threshold, the mapping block is determined to be a region of interest.

[0190] Calculate the area ratio of the motion area in the mapping block The specific calculation formula is as follows:

[0191] = (11)

[0192] in, is the total area of ​​the mapping block.

[0193] Determine whether a mapping block is an area of ​​interest based on the area ratio of the motion area in the mapping block:

[0194] (12)

[0195] in, A better detection effect can be obtained. In other embodiments, other values ​​can be taken, for example, obtained by the user re-analyzing according to the algorithm and his own coding example.

[0196] In the above embodiment, the second initialization threshold is obtained by looking up the table of the quantization parameter (Qp) input of the current image frame to be detected, so that different quantization parameters (Qp) index different second initialization thresholds, and the second initialization threshold can be adaptively adjusted according to the encoding information of the current image frame to be detected. This adaptive performance ensures the correctness of the detection of the region of interest of the current image frame to be detected, and detects the motion region of the current image frame to be detected based on the motion vector (MV) information of the encoded frame. This can be achieved by only caching these data, and the algorithm complexity is low.

[0197] In an exemplary embodiment, Figure 6 As shown, a video encoding method is provided, which is applied to Figure 1 The encoder module in FIG. 1 is taken as an example to illustrate the method, which includes the following steps S602 to S606. In which:

[0198] S602: Obtaining a region of interest obtained based on the image region of interest detection method in any one of the above embodiments.

[0199] The method for generating the region of interest can be found in the above description and will not be described in detail here. The encoder preprocessing module is used to identify the region of interest and does not affect the normal operation of the encoder module as a preprocessing module of the encoder module.

[0200] S604: Calculate and obtain a quantization parameter value of the current image frame based on the region of interest.

[0201] S606: Encode the image frame based on the quantization parameter value.

[0202] When encoding video frames through the region of interest, it is necessary to obtain a quantization parameter value. The quantization parameter value changes with the change of the region of interest. Therefore, the region of interest can be obtained with each current image frame to be detected, and then the quantization parameter offset value is calculated. Based on the quantization parameter offset value and the initial quantization parameter value, the quantization parameter value corresponding to the next image frame to be detected can be obtained for use in encoding the image frame.

[0203] In the above embodiment, the image region of interest detection algorithm is implemented in the encoder preprocessing module, which is only a pre-analysis stage of the formal encoder. The encoder preprocessing module and the formal encoder module are two parallel encoding processes, and the encoder preprocessing module does not affect the encoding performance of the formal encoder.

[0204] It should be understood that, although the steps in the flowcharts of the above embodiments are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the flowcharts of the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily to be performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0205] Based on the same inventive concept, embodiments of the present application further provide an image region of interest detection device and a video encoding device for implementing the aforementioned image region of interest detection method and the aforementioned video encoding method. The implementation solutions provided by these devices are similar to the implementation solutions described in the aforementioned methods. Therefore, the specific limitations in one or more of the following embodiments of the image region of interest detection device and the video encoding device can be found in the above-mentioned limitations of the image region of interest detection method and the video encoding method, and will not be further elaborated here.

[0206] In an exemplary embodiment, Figure 7 As shown, an image region of interest detection device is provided, comprising: an image acquisition module 701 to be detected, a mapping block division module 702, a texture region of interest detection module 703, a motion region of interest detection module 704 and a region of interest determination module 705, wherein:

[0207] The image acquisition module 701 is used to acquire the current image frame to be detected;

[0208] The mapping block division module 702 is configured to determine a mapping block size parameter value in the encoding parameters corresponding to the current image frame to be detected; and based on the mapping block size parameter value, divide the current image frame to be detected into a plurality of mapping blocks;

[0209] Texture region of interest detection module 703, used to perform texture region of interest detection on each mapping block to obtain a texture region of interest detection result;

[0210] A motion region of interest detection module 704 is configured to perform motion region of interest detection on each mapping block to obtain a motion region of interest detection result;

[0211] The region of interest determination module 705 is configured to determine the region of interest corresponding to the current image frame to be detected based on the texture region of interest detection result and the motion region of interest detection result.

[0212] In one embodiment, the above-mentioned texture region of interest detection module 703 is also used to divide each mapping block into sub-blocks and determine the size of the sub-blocks; obtain the first threshold adjustment factor corresponding to the current image frame to be detected; obtain the quantization parameter value of the current image frame to be detected; determine the first initialization threshold based on the quantization parameter value of the current image frame to be detected and the size of the sub-block; perform region of interest detection on the sub-block based on the first initialization threshold and the first threshold adjustment factor to obtain the texture region of interest detection result of the mapping block.

[0213] In one embodiment, the above-mentioned texture region of interest detection module 703 is also used to query the first mapping relationship table to determine the first initialization threshold corresponding to the quantization parameter value and the sub-block size; when the first initialization threshold corresponding to the quantization parameter value and the sub-block size is not found in the first mapping relationship table, based on the quantization parameter value and the sub-block size in the first mapping relationship table, the first initialization threshold corresponding to the quantization parameter value and the sub-block size is determined by interval size positioning.

[0214] In one embodiment, the texture region of interest detection module 703 is further configured to obtain an edge pixel judgment threshold based on a first threshold adjustment factor and a first initialization threshold; calculate an edge gradient value of a pixel in each sub-block; determine an edge pixel judgment result of the pixel based on the edge gradient value of the pixel and the edge pixel judgment threshold; determine an edge density value of the corresponding sub-block based on each edge pixel judgment result, and when the edge density value is greater than the edge density threshold, determine that the sub-block is a region of interest; and determine that the mapping block is a region of interest when the mapping block includes at least one sub-block that is a region of interest.

[0215] In one embodiment, the above-mentioned texture region of interest detection module 703 is also used to obtain a first initialization adjustment factor as a first threshold adjustment factor when the current image frame to be detected is an initial frame; when the current image frame to be detected is not an initial frame, obtain the first threshold adjustment factor based on the number of mapping blocks of the region of interest and the number of mapping blocks of the region of non-interest in the texture region of interest detection result of the previous frame of the image frame to be detected.

[0216] In one embodiment, the above-mentioned motion interest region detection module 704 is also used to obtain the quantization parameter value of the current image frame to be detected; obtain the second threshold adjustment factor corresponding to the current image frame to be detected; determine the second initialization threshold for different motion vector directions based on the quantization parameter value; perform motion detection on the prediction unit corresponding to each mapping block based on the second initialization threshold and the second threshold adjustment factor, and calculate the total area of ​​the motion region in each mapping block based on the motion detection result of each prediction unit; when the ratio of the total area of ​​the motion region in the mapping block to the area of ​​the mapping block is greater than the proportion threshold, the mapping block is determined to be the region of interest.

[0217] In one embodiment, the above-mentioned motion region of interest detection module 704 is also used to query the second mapping relationship table to determine the second initialization thresholds for different motion vector directions corresponding to the quantization parameter values; when the second initialization thresholds for different motion vector directions corresponding to the quantization parameter values ​​are not found in the second mapping relationship table, based on the quantization parameter values ​​in the second mapping table, the second initialization thresholds for different motion vector directions corresponding to the quantization parameter values ​​are determined by interval size positioning.

[0218] In one embodiment, the above-mentioned motion region of interest detection module 704 is further used to determine the motion vector threshold corresponding to each different motion vector direction based on the second initialization threshold and the second threshold adjustment factor of different motion vector directions; obtain the mapping block of the corresponding position of the previous frame of the image frame to be detected of the current image frame to be detected; when the motion vectors of the prediction units in different directions included in the obtained mapping block are all greater than the corresponding motion vector threshold, the prediction unit is determined to be a motion area; based on the area of ​​each prediction unit determined to be a motion area, the total area of ​​the motion area in the mapping block is obtained.

[0219] In one embodiment, the above-mentioned motion region of interest detection module 704 is also used to obtain a second initialization adjustment factor as a second threshold adjustment factor when the image frame to be detected is a target initial frame; when the image frame to be detected is not a target initial frame, obtain a second threshold adjustment factor based on the ratio of the area of ​​the motion region in the target image frame to be detected to the area of ​​the motion region in the previous frame of the image frame to be detected, wherein the target image frame to be detected is the frame before the previous frame of the image frame to be detected.

[0220] In one embodiment, the above-mentioned region of interest determination module 705 is also used to determine the interest type and interest intensity of each mapping block based on the texture region of interest detection results and motion region of interest detection results of the mapping block; and obtain the region of interest corresponding to the current image frame to be detected based on the interest type and interest intensity of each mapping block.

[0221] In one embodiment, the above-mentioned device also includes an output module for outputting the region of interest corresponding to the current image frame to be detected to the hardware quantization parameter value calculation module; or outputting the region of interest corresponding to the current image frame to be detected to the target storage area, and the region of interest corresponding to the current image frame to be detected in the target storage area is transmitted to the hardware encoder through the user interface.

[0222] In an exemplary embodiment, Figure 8 As shown, a video encoding device is provided, including: an area of ​​interest acquisition module 801, a quantization parameter value update module 802 and an encoding module 803, wherein:

[0223] A region of interest acquisition module 801 is configured to acquire a region of interest obtained based on the image region of interest detection device in any one of the above embodiments;

[0224] The quantization parameter value updating module 802 is configured to calculate the quantization parameter value of the current image frame based on the region of interest;

[0225] The encoding module 803 is configured to encode the image frame based on the quantization parameter value.

[0226] Each module in the above-mentioned image region of interest detection device and video encoding device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0227] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 9As shown. The computer device includes a processor, memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals via wired or wireless means, and the wireless means can be implemented via Wi-Fi, mobile cellular networks, NFC (near field communication), or other technologies. When executed by the processor, the computer program implements a method for detecting regions of interest in an image and a method for video encoding. The display unit of the computer device is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse.

[0228] Those skilled in the art will understand that Figure 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0229] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0230] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0231] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0232] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), data processing logic devices based on quantum computing, and the like.

[0233] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0234] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A method for detecting regions of interest in an image, characterized in that: The method is implemented in the encoder preprocessing module and is executed in parallel with the encoding process of the hardware encoder; the method includes: Get the current image frame to be detected; Determining a mapping block size parameter value in the encoding parameters corresponding to the current image frame to be detected, and dividing the current image frame to be detected into a plurality of mapping blocks based on the mapping block size parameter value; Performing texture region of interest detection on each of the mapping blocks to obtain a texture region of interest detection result; Performing motion region of interest detection on each of the mapping blocks to obtain a motion region of interest detection result; Determining a region of interest corresponding to the current image frame to be detected based on the texture region of interest detection result and the motion region of interest detection result; The performing texture region of interest detection on each of the mapping blocks to obtain a texture region of interest detection result includes: Dividing each of the mapping blocks into sub-blocks, and determining the size of the sub-blocks; Obtaining a first threshold adjustment factor corresponding to the current image frame to be detected; Obtaining a quantization parameter value of the current image frame to be detected; Determining a first initialization threshold based on the quantization parameter value of the current image frame to be detected and the size of the sub-block; Performing region of interest detection on the sub-block based on a first initialization threshold and the first threshold adjustment factor to obtain a texture region of interest detection result of the mapping block; The obtaining of a first threshold adjustment factor corresponding to the current image frame to be detected includes: When the current image frame to be detected is an initial frame, obtaining a first initialization adjustment factor as a first threshold adjustment factor; When the current image frame to be detected is not an initial frame, the number of mapping blocks of the area of ​​interest and the number of mapping blocks of the area of ​​non-interest in the texture area detection result of the previous image frame to be detected are obtained to obtain a first threshold adjustment factor; the higher the proportion of the area of ​​non-interest, the larger the first threshold adjustment factor, and the smaller the edge density value of the sub-block, the edge density value of the sub-block is determined based on the judgment result of each edge pixel of the sub-block, the edge pixel judgment result is obtained based on the pixel gradient value of the pixels in the sub-block and the edge pixel judgment threshold, and the edge pixel judgment threshold is obtained based on the first threshold adjustment factor and the first initialization threshold.

2. The method according to claim 1, characterized in that The determining of a first initialization threshold based on the quantization parameter value of the current image frame to be detected and the size of the sub-block includes: Querying a first mapping relationship table to determine a first initialization threshold corresponding to the quantization parameter value and the size of the sub-block; When the first initialization threshold corresponding to the quantization parameter value and the size of the sub-block is not found in the first mapping relationship table, the first initialization threshold corresponding to the quantization parameter value and the size of the sub-block is determined based on the quantization parameter value and the size of the sub-block in the first mapping relationship table by interval size positioning.

3. The method according to claim 1, characterized in that The performing the ROI detection on the sub-block based on the first initialization threshold and the first threshold adjustment factor to obtain a texture ROI detection result of the mapping block includes: Obtaining an edge pixel determination threshold based on a first threshold adjustment factor and the first initialization threshold; Calculating an edge gradient value of a pixel in each of the sub-blocks; and determining an edge pixel determination result of the pixel based on the edge gradient value of the pixel and the edge pixel determination threshold; Determining an edge density value of the corresponding sub-block based on each edge pixel judgment result, and determining that the sub-block is a region of interest when the edge density value is greater than an edge density threshold; When the mapping block includes at least one sub-block that is a region of interest, the mapping block is determined to be a region of interest.

4. The method according to any one of claims 1 to 3, characterized in that The performing motion region of interest detection on each of the mapping blocks to obtain a motion region of interest detection result of the mapping block includes: Obtaining a quantization parameter value of the current image frame to be detected; Obtaining a second threshold adjustment factor corresponding to the current image frame to be detected; determining second initialization thresholds for different motion vector directions based on the quantization parameter value; performing motion detection on the prediction units corresponding to the mapping blocks based on the second initialization threshold and the second threshold adjustment factor, and calculating the total area of ​​the motion region in each mapping block based on the motion detection results of the prediction units; When the ratio of the total area of ​​the motion regions in the mapping block to the area of ​​the mapping block is greater than a proportion threshold, the mapping block is determined to be a region of interest.

5. The method according to claim 4, characterized in that The determining, based on the quantization parameter value, second initialization thresholds for different motion vector directions includes: Querying a second mapping relationship table to determine second initialization thresholds for different motion vector directions corresponding to the quantization parameter values; When the second initialization thresholds for different motion vector directions corresponding to the quantization parameter values ​​are not found in the second mapping relationship table, the second initialization thresholds for different motion vector directions corresponding to the quantization parameter values ​​are determined based on the quantization parameter values ​​in the second mapping table by means of interval size positioning.

6. The method according to claim 4, characterized in that The performing motion detection on the prediction unit corresponding to each of the mapping blocks based on the second initialization threshold and the second threshold adjustment factor, and calculating the total area of ​​the motion region in each of the mapping blocks based on the motion detection result of each of the prediction units, includes: Determining a motion vector threshold corresponding to each different motion vector direction based on the second initialization threshold for each different motion vector direction and the second threshold adjustment factor; Obtaining a mapping block at a corresponding position of a previous image frame to be detected relative to the current image frame to be detected; When the motion vectors of the prediction unit included in the acquired mapping block in different directions are all greater than the corresponding motion vector threshold, determining that the prediction unit is a motion area; Based on the areas of the prediction units determined to be motion areas, the total area of ​​the motion areas in the mapping block is obtained.

7. The method according to claim 4, characterized in that The obtaining of the second threshold adjustment factor corresponding to the current image frame to be detected includes: When the image frame to be detected is a target initial frame, obtaining a second initialization adjustment factor as a second threshold adjustment factor; When the image frame to be detected is not the target initial frame, a second threshold adjustment factor is obtained based on the ratio of the area of ​​the motion area in the target image frame to be detected to the area of ​​the motion area in the previous image frame to be detected, wherein the target image frame to be detected is the previous frame of the previous image frame to be detected.

8. The method according to any one of claims 1 to 3, characterized in that The determining the region of interest corresponding to the current image frame to be detected based on the texture region of interest detection result and the motion region of interest detection result includes: determining an interest type and an interest intensity of each mapping block based on the texture ROI detection result and the motion ROI detection result of the mapping block; The region of interest corresponding to the current image frame to be detected is obtained based on the interest type and interest intensity of each mapping block.

9. The method according to any one of claims 1 to 3, characterized in that The method further comprises: Outputting the region of interest corresponding to the current image frame to be detected to a quantization parameter value calculation module; or The region of interest corresponding to the current image frame to be detected is output to a target storage area, and the region of interest corresponding to the current image frame to be detected in the target storage area is transmitted to a hardware encoder through a user interface.

10. A video encoding method, characterized in that: The video encoding method comprises: Obtaining a region of interest obtained by the method for detecting a region of interest in an image according to any one of claims 1 to 9; Based on the region of interest, a quantization parameter value of the current image frame is calculated; The image frame is encoded based on the quantization parameter value.

11. An image region of interest detection device, characterized in that: The device is implemented in an encoder preprocessing module, and the image region of interest detection process of the encoder preprocessing module is parallel to the encoding process of the hardware encoder; the device includes: The image acquisition module to be detected is used to obtain the current image frame to be detected; a mapping block division module, configured to determine a mapping block size parameter value in the encoding parameters corresponding to the current image frame to be detected; and based on the mapping block size parameter value, divide the current image frame to be detected into a plurality of mapping blocks; A texture region of interest detection module, configured to perform texture region of interest detection on each of the mapping blocks to obtain a texture region of interest detection result; A motion region of interest detection module, configured to perform motion region of interest detection on each of the mapping blocks to obtain a motion region of interest detection result; A region of interest determination module, configured to determine a region of interest corresponding to the current image frame to be detected based on the texture region of interest detection result and the motion region of interest detection result; The texture region of interest detection module is further configured to divide each of the mapping blocks into sub-blocks and determine the sizes of the sub-blocks; obtain a first threshold adjustment factor corresponding to the current image frame to be detected; obtain a quantization parameter value of the current image frame to be detected; determine a first initialization threshold based on the quantization parameter value of the current image frame to be detected and the size of the sub-block; perform region of interest detection on the sub-block based on the first initialization threshold and the first threshold adjustment factor to obtain a texture region of interest detection result for the mapping block; The texture region of interest detection module is also used to obtain a first initialization adjustment factor as a first threshold adjustment factor when the current image frame to be detected is an initial frame; when the current image frame to be detected is not an initial frame, obtain the first threshold adjustment factor based on the number of mapping blocks of the region of interest and the number of mapping blocks of the region of non-interest in the texture region detection result of the previous frame of the image frame to be detected; the higher the proportion of the region of non-interest, the larger the first threshold adjustment factor, and the smaller the edge density value of the sub-block, the edge density value of the sub-block is determined based on the judgment result of each edge pixel of the sub-block, the edge pixel judgment result is obtained based on the pixel gradient value of the pixel in the sub-block and the edge pixel judgment threshold, and the edge pixel judgment threshold is obtained based on the first threshold adjustment factor and the first initialization threshold.

12. A video device, characterized in that: The video device comprises: A region of interest acquisition module, configured to acquire a region of interest obtained based on the image region of interest detection device according to claim 11; A quantization parameter value updating module, configured to calculate a quantization parameter value of a current image frame based on the region of interest; The encoding module is configured to perform image frame encoding based on the quantization parameter value.

13. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 10 are implemented.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

Images