Coding method, medium and electronic device

Abstract

The present application relates to the field of video processing technology, and discloses a coding method, medium, and electronic device, which can improve the efficiency of video coding, avoid the waste of computing resources, and reduce video transmission delay, bandwidth, and bit rate. The coding method includes: obtaining an image to be processed; dividing the image to be processed into multiple blocks; detecting each block, and detecting a target area containing a target object in each block; encoding the image in the target area in each block according to a first coding parameter, and encoding the image in the area other than the target area in each block image according to a second coding parameter, wherein the image coding quality corresponding to the first coding parameter is higher than the image coding quality corresponding to the second coding parameter. The method is specifically used in the process of encoding videos such as high-definition live videos.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to an encoding method, a medium and an electronic device. Background Art

[0002] With the advent of 5G technology, ultra-high-definition video transmission has been booming in more and more industries, such as virtual reality (VR), augmented reality (AR), and live video. In the field of immersive video experience such as VR / AR, there are high requirements for resolution, real-time performance, and latency, such as resolution of 1080P, 4K, frame rate of 60FPS, 120FPS, and latency of about 30ms. Transmitting such a large amount of data has increased the demand for network bandwidth dramatically. Current video codec technologies such as H.265 use a series of technologies to compress the video before transmission, which can save a lot of bandwidth.

[0003] At present, video codec standards such as H.265 and AV1 have introduced tile block coding mechanisms, which independently and parallelly encode each tile block in the video frame to improve parallel processing capabilities and coding efficiency. Through tile block coding, only the block video of the area of ​​interest on the screen can be transmitted, reducing the requirements for video transmission bandwidth. In addition, based on the ROI (region of interest) coding mechanism, the ROI region of interest in the image is encoded with higher quality, and the area outside the ROI is encoded with lower quality, thereby saving bandwidth and controlling the bit rate. The acquisition of ROI generally uses visual algorithms such as the recent object detection (object detection) deep learning algorithm to perform model inference on the video frame to obtain the ROI of the object, such as the face area. Under the method of combining tile block coding and ROI coding, the ROI of an object may be divided into several different tile blocks. The part in the ROI area in each block is encoded with higher quality, and the part in other areas (except the ROI area) in each block image is encoded with lower quality.

[0004] However, each ROI region usually contains not only the foreground object in the image (i.e., the object of interest, such as the face, head, body, or animal of a person), but also the background at the edge of the object in the image. Especially when the size of the ROI region is large, the size of the background contained in the ROI region may be large. In this way, even if the video encoder uses the Tile block encoding mechanism and the ROI encoding mechanism to encode different blocks in each frame of the image, when the ROI region contains more background, the video codec will need to use more computing resources to encode these backgrounds with high quality, resulting in a waste of computing resources, and will also cause some problems such as waste of video transmission bandwidth and increased latency. Summary of the invention

[0005] The embodiments of the present application provide a coding method, a medium and an electronic device, which can improve the efficiency of video coding, avoid the waste of computing resources, reduce video transmission delay, and lower bandwidth and bit rate.

[0006] In a first aspect, an embodiment of the present application provides an encoding method, including: acquiring an image to be processed; dividing the image to be processed into multiple blocks; detecting each block to detect a target area containing a target object in each block; encoding the image in the target area in each block according to a first encoding parameter, and encoding the image in an area other than the target area in each block image according to a second encoding parameter, wherein the image encoding quality corresponding to the first encoding parameter is higher than the image encoding quality corresponding to the second encoding parameter.

[0007] The blocks in the above-mentioned image to be processed may be Tile blocks. The detection of each block may be specifically a region of interest (ROI) detection, and the target region containing the target object in each block is the ROI region in each block. As an example, when the image to be processed contains a person, the target object may be the head of the person, such as Figure 1 and Figure 2 The target object in the image to be processed may be a person's head including a hat. It is understandable that the target object in the image to be processed is segmented into various blocks, and the target object contained in each block refers to a portion of the target object contained in each block, that is, a segment containing the target object.

[0008] In some embodiments, the quantization parameter represented by the first encoding parameter is less than the quantization parameter represented by the second encoding parameter. For example, for luminance (Luma) encoding, the quantization parameter (quantization parameter, QP) takes a value of 0 to 51, and for chrominance (Chroma) encoding, the QP takes a value of 0 to 39. It can be understood that the quantization parameter QP reflects the compression of spatial details. The smaller the value, the finer the quantization, the higher the image quality, and the longer the generated code stream. In some embodiments, the first encoding parameter (such as encoding parameter 1 below) is a value with a smaller QP, corresponding to a higher quality encoding of the ROI area, and the second encoding parameter (such as encoding parameter 2 below) is a value with a larger QP, corresponding to a lower quality encoding of the ROI area. In this way, the encoding method provided in the embodiment of the present application is to detect each block separately to obtain the ROI area in each block, so that the ROI area on each block contains less background other than the target object, thereby avoiding high-quality encoding of the background in the image as much as possible, improving the encoding efficiency, and reducing the waste of computing resources during the encoding process.

[0009] In a possible implementation of the first aspect, the above-mentioned detection of each block to detect the target area containing the target object in each block includes: detecting the image to be processed to detect the original area containing the target object in the image to be processed (the original ROI area mentioned below, such as ROI area a); dividing the original area into multiple blocks to obtain multiple sub-areas (such as sub-ROI areas mentioned below); detecting the image in the sub-area in each block (such as ROI detection) to detect the target area (ROI area) containing the target object in each block. It can be understood that by performing detection on the image in the sub-ROI area of ​​each block, the background contained in the sub-ROI area except the target object can be further removed to obtain the final target area on each block, so that the target area on each block contains as little background as possible.

[0010] In a possible implementation of the first aspect, the above-mentioned detection of the image to be processed and the detection of the original area containing the target object in the image to be processed include: adjusting the image to be processed to a first image size and inputting it into a first object detection model (i.e., the first coarse adjustment object detection model hereinafter), and the first object detection model outputs the original area in the image to be processed; in addition, the above-mentioned detection of the image in the sub-area in each block and the detection of the target area containing the target object in each block include: adjusting the image in the sub-ROI area in each block to a second image size and inputting it into a second object detection model (i.e., the second fine adjustment object detection model hereinafter), and the second object detection model outputs the target area in each block. Wherein, the first image size is larger than the second image size. The model structure of the above-mentioned first object detection model and second object detection model can adopt open source model structures such as YOLO, MobileNet-SSD and CornetNet, or adopt a specially designed model structure. It can be understood that the first object detection model is used to detect the object ROI on the whole image, and then the image of this ROI is sent to the secondary fine-tuning object detection model after being divided by Tile, so as to narrow the detection range of the secondary fine-tuning object detection model. The images of ROIs in multiple blocks can be stacked into batches and sent to the second object detection model for reasoning tasks, which can make full use of the computing resources of the NPU hardware and improve the reasoning efficiency. In addition, the two-level object detection model can detect a more accurate ROI in each block, so that the ROI in each block contains as little background as possible except for the target object of interest.

[0011] In a possible implementation of the first aspect, the above-mentioned image to be processed is any frame of the video to be processed; when the image to be processed is an I frame in the video to be processed, each block in the image to be processed is encoded based on intra-frame coding according to the first coding parameter and the second coding parameter; when the image to be processed is a P frame or a B frame in the image to be processed, each block in the image to be processed is encoded based on inter-frame coding according to the first coding parameter and the second coding parameter. It can be understood that for a frame of the image, the image in the target area (i.e., the ROI area) finally calculated is encoded according to the first coding parameter, and the area in the image other than the target area is encoded according to the second coding parameter. In this way, it is helpful to further improve the coding efficiency of the video.

[0012] In a possible implementation of the first aspect, the method further includes: when encoding the video to be processed as a video code stream based on the first encoding parameter and the second encoding parameter, sending the video code stream to the cloud device; wherein the cloud device is used to send the video code stream to the receiving device, and the receiving device is used to decode the video code stream and play the video to be processed. For example, the video to be processed is a video in a live scene. It can be understood that encoding the video to be processed based on the first encoding parameter and the second encoding parameter means that each block in each frame of the video to be processed is encoded in parallel, and specifically, each block is encoded according to the first encoding parameter for the image in the ROI area (i.e., the target area), and the image in the area other than the ROI area is encoded according to the second encoding parameter. In this way, in the encoding method provided in the embodiment of the present application, while the encoding process of a frame of video image adopts Tile parallel encoding, the ROI area that focuses more on the object is used to encode different parts on each block using suitable encoding parameters. Thus, it is possible to avoid high-quality encoding of too much background in each frame of the image, avoid waste of computing resources, improve the encoding efficiency of the video, and thus reduce video transmission delay, bandwidth requirements, and transmission bit rate. This makes the live broadcast of high-definition video scenes smooth, clear and with low latency.

[0013] In the second aspect, the present application provides an electronic device, including: an acquisition module for acquiring an image to be processed; a division module for dividing the image to be processed acquired by the acquisition module into multiple blocks; a detection module for detecting each of the blocks divided by the division module, and detecting a target area containing a target object in each of the blocks; an encoding module for encoding the image in the target area of ​​each block detected by the detection module according to a first encoding parameter, and encoding the image in the area other than the target area in each block image according to a second encoding parameter, wherein the image encoding quality corresponding to the first encoding parameter is higher than the image encoding quality corresponding to the second encoding parameter. In some embodiments, the acquisition module can be implemented by an ISP in an electronic device, the division module can be implemented by a CPU in an electronic device, the detection module can be implemented by an NPU in an electronic device, and the encoding module can be implemented by a VPU in an electronic device. For example, the electronic device can be the high-definition video live broadcast acquisition device 31 described below, the detection module can be an NPU 313, and the encoding module can be a VPU 312.

[0014] In a possible implementation of the second aspect above, the detection module is specifically used to detect the image to be processed, detect the original area containing the target object in the image to be processed; divide the original area into multiple blocks to obtain multiple sub-areas; detect the image in the sub-area in each block, and detect the target area containing the target object in each block.

[0015] In a possible implementation of the second aspect, the detection module is specifically used to adjust the image to be processed to a first image size and input it into a first object detection model, and the first object detection model outputs the original area in the image to be processed; and adjust the image in the sub-ROI area in each block to a second image size and input it into a second object detection model, and the second object detection model outputs the target area in each block.

[0016] In a possible implementation of the second aspect above, the above-mentioned image to be processed is any frame image in the video to be processed; when the image to be processed is an I frame in the video to be processed, each block in the image to be processed is encoded based on intra-frame coding according to the first coding parameter and the second coding parameter; when the image to be processed is a P frame or a B frame in the image to be processed, each block in the image to be processed is encoded based on inter-frame coding according to the first coding parameter and the second coding parameter.

[0017] In a possible implementation of the second aspect, the device further includes: a sending module, which is used to send a video code stream to a cloud device when the video to be processed is encoded as a video code stream based on the first encoding parameter and the second encoding parameter; wherein the cloud device is used to send the video code stream to a receiving device, and the receiving device is used to decode the video code stream and play the video to be processed, and the video to be processed is a live video. For example, the sending module can be implemented by a network module in the electronic device, such as when the electronic device is the high-definition video live broadcast acquisition device 31 described below, the sending module can be a network module 314.

[0018] In a third aspect, the present application provides a readable medium having instructions stored thereon, which, when executed on an electronic device, causes the electronic device to execute the encoding method in the first aspect and any possible implementation thereof.

[0019] In a fourth aspect, an electronic device is provided in the implementation of the present application, comprising: a memory for storing instructions executed by one or more processors of the electronic device, and a processor, which is one of the processors of the electronic device, for executing the encoding method in the first aspect and any possible implementation thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 According to some embodiments of the present application, a schematic diagram of blocks and ROI regions in an image is shown;

[0021] Figure 2 According to some embodiments of the present application, a schematic diagram of blocks and ROI regions in an image is shown;

[0022] Figure 3 According to some embodiments of the present application, a schematic diagram of a video encoding scenario is shown;

[0023] Figure 4 According to some embodiments of the present application, a schematic diagram of the structure of a high-definition video live broadcast acquisition device is shown;

[0024] Figure 5 According to some embodiments of the present application, a schematic diagram of an encoding method is shown;

[0025] Figure 6 According to some embodiments of the present application, a schematic diagram of an encoding method is shown. DETAILED DESCRIPTION

[0026] Illustrative embodiments of the present application include, but are not limited to, encoding methods, apparatus, media, and devices.

[0027] For the convenience of description, some nouns or terms in the embodiments of the present application are first explained.

[0028] 1. Tile:

[0029] An image can be divided into several tiles, that is, an image is divided into several rectangular areas in the horizontal and vertical directions, and each rectangular area is a tile. The main purpose of dividing tiles is to enhance the parallel processing capability of video encoding and control the bit stream. Only the block video encoding data of interest can be transmitted, and there is no need for complex thread synchronization when using it. The division of tiles does not require uniform distribution of horizontal and vertical boundaries, and can be flexibly controlled according to the requirements of parallel computing and error control.

[0030] 2. Region of interest (ROI):

[0031] In the field of image processing, a region of interest (ROI) is an image area selected from an image. This area is the focus of image analysis. Specifically delineating the area in order to further process the target object in the area (such as a person's face, head, body, or animal, etc.) can reduce processing time and increase accuracy. Among them, ROI can be an area outlined from the image to be processed in the form of a rectangle, circle, ellipse, irregular polygon, etc. For example, in and Various object detection algorithms are often used in machine vision software to obtain the ROI of an image, or deep learning algorithms are used to detect the ROI of an image, and then the image is processed in the next step. Among them, the ROI area is usually a rectangle, and the target object selected by the ROI area frame is usually an irregular shape. Therefore, the original rectangular ROI area contains not only the target object but also some background around the target object (that is, the background where the inner edge of the rectangular frame is located).

[0032] 3. Intra-frame coding:

[0033] Intra-frame coding is spatial domain coding, which uses the spatial redundancy of images to compress images. It processes an independent image and does not span multiple images. Spatial domain coding relies on the similarity between adjacent pixels in an image and the main spatial domain frequency of the pattern area.

[0034] 4. Inter-frame coding:

[0035] Inter-frame coding is a temporal coding that uses the temporal redundancy between a set of consecutive images to compress images. If a frame of an image can be used by the decoder, then the decoder only needs to use the difference between two frames of images to get the next frame of the image. For example, the similarities between several frames of images with gentle motion are large and the differences are small, while the similarities between several frames of images with violent motion are small and the differences are large. Temporal coding relies on the similarity between consecutive image frames, and uses the image information that has been received and processed as much as possible to "predict" and generate the current image.

[0036] 5. Compression algorithm:

[0037] As an example, the video compression algorithm includes steps 1)-4).

[0038] 1) Grouping: Group several frames of images into a group of pictures (GOP). To prevent motion changes, the number of frames should not be too large. A GOP is a group of continuous pictures, that is, a sequence.

[0039] 2) Define frames: Define each frame image in each group into three types, namely I frame, B frame and P frame.

[0040] 3) Prediction frame: Use I frame as base frame, use I frame to predict P frame, and then use I frame and P frame to predict B frame.

[0041] 4) Data transmission: Finally, the I frame data and the predicted difference information are stored and transmitted.

[0042] Among them, I frame is also called intra-frame coded frame (also called key frame), P frame is also called inter-frame prediction coded frame (forward reference frame), and B frame is also called bidirectional prediction coded frame (bidirectional reference frame).

[0043] The encoding method provided in this application can be applied to fields such as automobiles, embedded devices, video surveillance, smart homes, medical care, and the Internet of Things.

[0044] More specifically, the above encoding method can be applied to some full HD video transmission scenarios, such as compression encoding of video images involved in virtual reality (VR), augmented reality (AR), mixed reality (MR), 1080P, 4K and other video transmission scenarios.

[0045] The embodiments of the present application will be described in further detail below in conjunction with the accompanying drawings.

[0046] Typically, in ROI-based video encoding scenarios, conventional encoding methods use AI object detection algorithms to independently detect the ROI area of ​​a complete frame of an image, which can be called the original ROI area. When using the Tile block encoding mechanism to encode different blocks (i.e., Tiles) in a frame of an image in parallel, the original ROI area is divided into different blocks, and the portion of each block that is in the original ROI area is encoded with high quality, and the portion of each block that is not in the original ROI area is encoded with low quality. It is understandable that the image after low-quality encoding has lower quality than the image after high-quality encoding, such as lower resolution.

[0047] As an example, Figure 1 As shown, the image A is divided into four blocks A1, A2, A3 and A4 of two rows and two columns by the Tile block coding mechanism. Assume that the target object in the image A is a person's head (including the person's hat and face). The above conventional coding method is to identify the complete person's head in the image A, and then determine the ROI area a (original ROI area) corresponding to the complete person's head. In other words, the ROI area a focuses on the complete person's head in the image A. Subsequently, the above conventional coding method can directly divide the ROI area a into blocks A1, A2, A3 and A4, and obtain ROI areas a1, a2, a3 and a4 respectively, and then respectively perform high-quality coding on the images in the ROI areas a1, a2, a3 and a4 in each block, that is, all parts of the image A in the original ROI area a are encoded with high quality.

[0048] However, since the target object (i.e., the object to be focused) in an image is usually irregular in shape, and the ROI area is usually a regular shape such as a rectangle, in order to make the encoded image contain as many features of the target object as possible, the ROI area needs to contain all edge points of the target object. For example, Figure 1 The lower edge of the hat of the person shown is longer than the person's chin, and the right edge of the hat is longer than the person's right face, and the ROI area a is a rectangle. In order to make the encoded image include the complete head of the person, the ROI area a needs to include more background outside the head of the person, such as more background under the chin and more background on the right side of the right face.

[0049] As an example, Figure 1 The object segment (i.e., part of the head) on the block A4 shown is part of the target object (i.e., the entire head of the person), so the lower border of the ROI area a4 in the block A4 exceeds the straight line n where the chin edge point is located, and the right border of the ROI area a4 exceeds the straight line q where the right face edge point is located, that is, the ROI area a4 contains more background outside the face.

[0050] In this way, the conventional ROI video encoding method is used to perform ROI encoding on the video, so that a part of the background is included in the ROI area, and high-quality encoding is also used for this part of the background, resulting in problems such as waste of video bandwidth and high transmission bit rate.

[0051] In order to solve the above problem, an encoding method provided in an embodiment of the present application performs ROI detection on each block when the video encoder uses the tile encoding mechanism to encode different blocks (i.e., tiles) in a frame of an image. That is, the AI ​​object detection algorithm is performed on each block to obtain the ROI area. When performing ROI detection on a block, the area where the object of interest (i.e., the target object) in the block is detected. For example, assuming that Figure 1 To perform ROI detection on block A4 in the image, only the area where the head of the person is located needs to be detected. The edges of the ROI areas on each block are closely attached to the edges of the head of the person in each block, thereby obtaining a more accurate ROI area in each block, so that the image in the ROI area of ​​each block contains less background except for the target object. Furthermore, when the video encoder performs high-quality encoding on the ROI area in each block, the image in the ROI area in each block contains less background, so compared with Figure 1 The encoding scheme shown, the encoding method provided by the present application can avoid high-quality encoding of too much background, thereby improving encoding efficiency, avoiding waste of computing resources, and having certain benefits in reducing video transmission delay, reducing bandwidth and bit rate.

[0052] As an example, Figure 2 As shown, the encoding method provided in the embodiment of the present application can divide the image A into four blocks A1, A2, A3 and A4 of two rows and two columns, detect ROI areas a1, a2, a3 and a4 respectively, and then detect ROI areas b1, b2, b3 and b4 respectively based on the ROI areas a1, a2, a3 and a4.

[0053] Specifically, in an embodiment of the present application, the ROI-based AI object detection algorithm adopts a two-level model mechanism, including a first-level coarse-tuning object detection model and a second-level fine-tuning object detection model, to realize ROI detection from coarse to fine.

[0054] It is understandable that the encoding method provided in the present application designs a two-level neural network object detection model, including a first-level coarse adjustment object detection model (i.e., a first-level coarse adjustment object detection model or a first-level model) and a second-level fine adjustment object detection model (i.e., a second-level fine adjustment object detection model). Among them, the first-level coarse adjustment object detection model aims to obtain a rough object edge frame ROI on the original video, and the second-level fine adjustment object detection model further fine-tunes the ROI obtained by the first-level coarse adjustment object detection model to obtain an accurate ROI.

[0055] In addition, the model structure in the above-mentioned 2-level model mechanism can adopt open source model structures such as YOLO, MobileNet-SSD and CornetNet, or adopt a specially designed model structure. The training data set can adopt an open source data set, or self-collected data and annotated data. During the training process, the model converges on the data set and the detection accuracy meets the requirements of the application.

[0056] Specifically, Figure 2 As shown, for image A, the encoding method provided by the embodiment of the present application is as follows: the original image is adjusted to a fixed size such as 300*300 (pixels), and sent to a first-level coarse-tuning object detection model to obtain the ROI rectangular frame of the target object, that is, the ROI area; image A is divided into different blocks according to the pre-configured tile block parameters, such as dividing image A into four blocks A1, A2, A3 and A4 with two rows and two columns; then, the image contained in the ROI area detected by the first-level object detection model in each block (i.e., ROI areas a1, a2, a3, a4) is adjusted to a fixed size such as 48*48 (pixels), and then sent to a second-level fine-tuning object detection model to obtain a more accurate ROI area. In this way, the ROI area b1 on block A1, the ROI area b2 on block A2, the ROI area b3 on block A3, and the ROI area b4 on block A4 are detected.

[0057] It is understandable that when a frame of image is divided into multiple blocks by using the Tile block encoding mechanism, the target object in the image is divided into multiple blocks, which usually generates multiple segments (ie, segments of the target object).

[0058] As an example, Figure 2 The head of the person shown (ie, the target object) is divided into blocks A1, A2, A3, and A4 to form a segment respectively. For example, the right side of the lower half of the face of the person's head is a segment divided into block A4.

[0059] For example, as shown in Figure 2, the segment on block A4 is a face from the nose to the chin, and the ROI region b4 focusing on the segment contains less background other than the face. Specifically, Figure 2The lower border of the middle ROI region b4 exceeds the straight line n where the edge point of the chin is located less, and the right border exceeds the straight line q where the edge point of the right face is located less. Obviously, Figure 2 The ROI area b4 shown is compared to Figure 1 The ROI region a4 shown contains less background other than the face of the person.

[0060] Similarly, comparing ROI regions b1, b2, b3 with regions a1, a2, a3 respectively, ROI regions b1, b2, b3 and b4 all contain less background other than the face, that is, ROI regions b1, b2, b3 and b4 as a whole contain less background than ROI region a.

[0061] In this way, the encoding method provided in the embodiment of the present application can detect each block in the image separately to obtain the ROI area in each block, so that the ROI area on each block in the image contains less background other than the segment of the target object, thereby avoiding high-quality encoding of the background in the image as much as possible, improving encoding efficiency, and reducing the waste of computing resources in the encoding process.

[0062] In some embodiments, the encoding method provided in the embodiments of the present application can be applied to electronic devices that support video compression formats such as the H.265 standard and the AV1 standard. More specifically, in some embodiments, electronic devices applicable to the present application include, but are not limited to: high-definition video live broadcast acquisition devices, televisions or set-top boxes, wearable devices, AR devices, VR devices, MR devices, digital direct broadcast systems, wireless communication devices, personal digital assistants (PDAs), laptop or desktop computers, mobile phones, tablet computers, digital cameras, digital recording devices, video game devices, cellular or satellite radio phones, etc.

[0063] In the following embodiments, an electronic device that executes the encoding method in the application is taken as an example to illustrate a high-definition video live broadcast acquisition device.

[0064] like Figure 3 As shown, a video encoding and decoding scenario provided in an embodiment of the present application includes a high-definition video live broadcast acquisition device 31 and a high-definition video live broadcast terminal device 32.

[0065] The high-definition video live broadcast acquisition device 31 acquires high-definition video, such as 1080P, 4K, encodes the acquired video to obtain a video code stream, and distributes the video code stream to the high-definition video live broadcast terminal device 32 through a network such as the cloud, and then the high-definition video live broadcast terminal device 32 decodes and displays the video code stream.

[0066] Specifically, the high-definition video live broadcast acquisition device 31 includes an image processor (Image Signal Processor, ISP) 311, a video codec (Video Processing Unit, VPU, also known as a video processor) 312 and a neural network processor (Neural-network Processing Unit, NPU) 313, a network module 314, a memory 315, a display unit 316, a central processing unit (central processing unit, CPU) 317 and a camera 318.

[0067] Among them, the camera 318 is used to capture images. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP 311 to convert it into a digital image signal. It can be understood that the high-definition video live broadcast acquisition device 31 can realize the video shooting function through the ISP 311, the camera 318, the VPU 312, the display unit 316, etc.

[0068] VPU 312 is a core engine of a video processing platform, which has hardware encoding and decoding functions and can reduce the CPU load of the device. For example, in an embodiment of the present application, VPU 312 is used to encode the video to obtain a video code stream to implement video compression encoding.

[0069] NPU 313 is an architecture designed for deep learning reasoning and can be used to process multimedia data such as video and images. For example, in this application, NPU is used to detect the ROI area of ​​the object using AI technology, including the object ROI of the complete image and the object ROI of each block (i.e., the ROI area of ​​the object segment).

[0070] The display unit 316 is used to display a human-computer interaction interface, images, videos, etc., for example, display a video captured by the camera 318 .

[0071] The memory 315 is used to store instructions and data executed by processors such as CPU 317, ISP 311, VPU 312, NPU 313, etc. during the execution of the encoding method.

[0072] The CPU 317 is used to encode the video in cooperation with the ISP 311, the VPU 312, the NPU 313, the memory 315 and other units. As an example, the CPU 317 is used to control the ISP 311, the VPU 312, and the NPU 313 to execute the video encoding method.

[0073] The network module 314 is used to send the encoded video code stream to the cloud. The network module 314 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), mobile communication networks, etc. applied to the high-definition video live broadcast acquisition device 31. The network module 314 can communicate with the network and other devices through wireless communication technology. Among them, the cloud device on the cloud can be a server, etc.

[0074] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the high-definition video live broadcast acquisition device 31. In other embodiments of the present application, the high-definition video live broadcast acquisition device 31 may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. For example, the high-definition video live broadcast acquisition device 31 may also include a sensor module, an audio module, etc. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0075] In some embodiments, Figure 4 As shown, it is a possible structural diagram of the high-definition video live broadcast acquisition device 31. Figure 4 The device shown includes a network module 314, a video codec (VPU) 312, a neural network processor (NPU) 313, a display unit 316, a central processing unit (CPU) 317 and a memory 315, as well as a bus 319 connecting these devices.

[0076] The high-definition video live broadcast terminal device 32 includes a network module 321, a VPU 322, a memory 323, a display unit 324 and a CPU 325.

[0077] The network module 321 receives the video code stream from the high-definition video live broadcast acquisition device 31 through the cloud, and the video code stream is decoded by the VPU 322 and then broadcasted by the display unit 324.

[0078] The memory 323 is used to store instructions executed by the CPU 325, VPU 322 and other processors during the encoding method. The CPU 325 is used to cooperate with the VPU 322, the memory 323 and other units to decode and play the video.

[0079] It is understandable that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the high-definition video live broadcast terminal device 32.

[0080] The following will introduce the workflow of the encoding method executed by each unit in the HD video live broadcast acquisition device 31 based on some embodiments of the present application, combined with the description of the above scenario and the description of the HD video live broadcast acquisition device 31, taking the encoding process of a frame of image as an example. The technical details described in the above scenario are still applicable in this process, and some will not be repeated here to avoid repetition. Figure 5 As shown, it is a flow chart of a method for encoding a frame of image provided by an embodiment of the application, and the method specifically includes:

[0081] Step 501: The high-definition video live broadcast acquisition device 31 pre-configures the parameters of Tile blocks, such as the parameters of Tile blocks are N rows and M columns, which means that an image is divided into Tile blocks of N rows and M columns.

[0082] For example, combining the above Figure 2 For image A, the pre-configured Tile partitioning parameters may be two rows and two columns (N=M=2), that is, image A may be divided into four partitions A1, A2, A3 and A4. Each partition is a sub-image, also called a partitioned sub-image.

[0083] In some embodiments, for a frame of image, the parameters of the Tile blocks may include the coordinates of each block relative to the image, such as the coordinates of the four vertices of each rectangular block on the image.

[0084] As an example, the above step 501 can be executed by the CPU 317 in the high-definition video live broadcast acquisition device 31.

[0085] Step 502: The high-definition live video acquisition device 31 runs a first-level coarse-tuning object detection model through the NPU 313, inputs the entire image, and infers ROI-A.

[0086] It can be understood that ROI-A is a roughly estimated ROI area on the entire image input to the first-level coarse adjustment object detection model. Figure 2 For example, when the first-level coarse-tuning object detection model inputs image A, the ROI area a (i.e., ROI-A) is inferred.

[0087] by Figure 2 For example, before inputting image A into the first-level coarse-tuning object detection model, NPU 313 or ISP 311 may also adjust image A to a fixed size, such as 300*300.

[0088] In some embodiments, when the NPU 313 detects that the ROI region of the entire image to be processed is a rectangle, the position of an ROI region on the image may be identified by the coordinates of four vertices of each rectangular ROI region.

[0089] Step 503: the high-definition video live broadcast acquisition device 31 divides ROI-A into multiple sub-ROIs according to the configured Tile blocks, and extracts the image of each sub-ROI.

[0090] As an example, the above step 503 can be executed by devices such as the CPU 317 and the ISP 311 in the high-definition video live broadcast acquisition device 31.

[0091] For example, combining the above Figure 2 For image A, the high-definition video live broadcast acquisition device 31 directly divides the ROI area a (i.e., ROI-A) into blocks, and obtains the ROI areas a1, a2, a3, and a4 corresponding to the blocks A1, A2, A3, and A4, respectively. At this time, the ROI areas a1, a2, a3, and a4 are the above-mentioned multiple sub-ROIs. Then, the high-definition video live broadcast acquisition device 31 can extract the images in the ROI areas a1, a2, a3, and a4 in image A, respectively.

[0092] ROI regions b1, b2, b3 and b4 are detected on the images in ROI regions a1, a2, a3 and a4, respectively, that is, ROI regions where target object segments are gathered on each block are detected. Accordingly, the NPU 312 can use other regions on blocks A1, A2, A3 and A4 except the ROI regions (i.e., ROI regions b1, b2, b3 and b4) as non-ROI regions.

[0093] In some embodiments, when the NPU 313 detects that the sub-ROIs on each block of the image to be processed are rectangular, the position of an ROI area on the corresponding block can also be identified by the four vertex coordinates of the ROI area of ​​each rectangle. Of course, these coordinates are relative to the block or relative to the image.

[0094] It is understandable that in the embodiment of the present application, when a frame of image includes multiple target objects (such as multiple human heads), multiple ROI-A can be detected through the first-level coarse-tuning object detection model. Correspondingly, some blocks of the image may include segments of different target objects, so sub-ROIs belonging to different ROI-A on these blocks can be divided, that is, sub-ROIs where segments of different target objects are located. For example, a frame of image includes person 1 and person 2, and the first-level coarse-tuning object detection model can be used to detect the ROI-A where person 1 is located and the ROI-A where person 2 is located. At this time, if a block on the image includes both segments of person 1 and segments of person 2, sub-ROIs of the two segments in different ROI-A can be divided.

[0095] Step 504: The high-definition live video acquisition device 31 adjusts the image of each sub-ROI to the same size and stacks them up as input to the secondary fine-tuning object detection model. The NPU 313 runs the secondary fine-tuning object detection model to infer the accurate ROI-B on the image of each sub-ROI.

[0096] by Figure 2 For example, the high-definition video live broadcast acquisition device 31 can adjust the images in the ROI areas a1, a2, a3 and a4 in the image A to the same size, such as 48*48. In addition, the secondary fine-tuning object detection model can detect the ROI areas b1, b2, b3 and b4 corresponding to the blocks A1, A2, A3 and A4, that is, the accurate ROI-B on each block, based on the images in the ROI areas a1, a2, a3 and a4.

[0097] The area except ROI-B on each block of the image to be processed is called a non-ROI area. Then, the area except ROI-B on all blocks in the image to be processed is a non-ROI area in the image.

[0098] In some embodiments, before using the primary coarse-tuning object detection model and the secondary fine-tuning object detection model, the embodiment of the present application trains the deep learning model with a pre-collected and annotated training data set, so that the model can perform the above steps. The images in the above training data set can be of a preset size (such as 100*100 or 48*48) and pre-marked with ROI.

[0099] Specifically, in some embodiments, the images in the training data set are input into the object detection deep learning model (such as the first-level coarse-tuning object detection model and the second-level fine-tuning object detection model), and the model inference outputs the ROI position and score (confidence, etc.) information. The model loss function can be the sum of the positioning loss and the classification loss (for details, please refer to MobileNet-SSD). The model is continuously fed with new data sets for training. After multiple iterations, until the model converges, the loss function stabilizes at a relatively low value, and the accuracy on the evaluation data set reaches the expected goal.

[0100] As an example, the object detection deep learning model can use open source models such as YOLO, MobileNet-SDD, CornerNet, etc., or a specially designed network architecture. Among them, the training data set of the first-level coarse-tuning object detection model can use an open source data set, such as the WIDER FACE data set for face detection. You can also use your own collected data set and mark the object box ROI. The training data set of the second-level fine-tuning object detection model can use the training data set of the first-level coarse-tuning object detection model. The entire image is divided into random tiles, and the marked ROI also needs to be divided according to the tile blocks. In addition, in order to obtain a more accurate ROI, the marked ROI can be manually marked again after being segmented to obtain an accurate ROI area.

[0101] Step 505: The high-definition video live broadcast acquisition device 31 encodes the image in ROI-B in each Tile block in the image according to encoding parameter 1 through VPU 312, and encodes the image in the non-ROI area in each Tile block according to encoding parameter 2.

[0102] Among them, the non-ROI area in each Tile block is the area in the image except the ROI-B area.

[0103] It can be understood that the encoding parameter 1 may be an encoding parameter corresponding to the ROI area, and is used to encode the image data with high quality, that is, the encoding parameter 1 corresponds to a higher image encoding quality. The encoding parameter 2 may be an encoding parameter corresponding to the non-ROI area, and is used to encode the image data with lower quality, that is, the encoding parameter 2 corresponds to a lower image encoding quality.

[0104] It can be understood that adjusting the coding parameters is mainly to adjust the quantization parameter (QP). For luminance (Luma) coding, QP takes values ​​0 to 51, and for chrominance (Chroma) coding, QP takes values ​​0 to 39. Among them, the quantization parameter QP reflects the compression of spatial details. The smaller the value, the finer the quantization, the higher the image quality, and the longer the generated code stream. If the QP is small, most of the details in the image will be retained; when the QP increases, some details are lost, the bit rate is reduced, but the image distortion is enhanced and the quality is reduced. For example, for luminance coding, when QP takes the minimum value of 0, it means that the quantization is the finest; on the contrary, when QP takes the maximum value of 51, it means that the quantization is the coarsest. Of course, the quantization parameters represented by the coding parameters provided in the embodiments of the present application include but are not limited to the above examples.

[0105] In some embodiments, encoding parameter 1 is a smaller QP value, corresponding to higher quality encoding of the ROI area, and encoding parameter 2 is a larger QP value, corresponding to lower quality encoding of the ROI area. In other words, the quantization parameter represented by encoding parameter 1 is smaller than the quantization parameter represented by encoding parameter 2. The selection of the QP value can be determined according to the actual bit rate requirement and system bandwidth requirement.

[0106] It can be understood that the first-level coarse-tuning object detection model is used to detect the object ROI on the overall image, and then the image of this ROI is sent to the second-level fine-tuning object detection model after being divided by Tile, so as to narrow the detection range of the second-level fine-tuning object detection model. The images of ROIs in multiple tile blocks can be stacked into batches, and when sent to the second-level fine-tuning object detection model for reasoning tasks, the computing resources of the NPU hardware can be fully utilized to improve the reasoning efficiency. In addition, the two-level object detection model can detect a more accurate ROI in each Tile block, so that the ROI in each block contains as little background as possible except for the target object of interest. Thereby, high-quality encoding of too much background can be avoided, thereby improving encoding efficiency, avoiding waste of computing resources, and having certain benefits in reducing video transmission delay, bandwidth and bit rate.

[0107] Generally speaking, for video encoding, there are mainly intra-frame encoding (I frame) and inter-frame encoding (P / B frame). The primary and secondary object detection models can act on each original frame to obtain the precise ROI corresponding to each original frame. If the current frame is processed as an I frame, it is encoded according to the ROI corresponding to the I frame and the current frame. If the current frame is processed as a P / B frame, it is encoded according to the ROI corresponding to the P / B frame and the current frame.

[0108] Further, see Figure 5 The encoding method shown in the figure, the following takes the encoding process of multiple frames of video images in a video as an example to introduce the workflow of each unit in the high-definition video live acquisition device 31 to execute the encoding method. Figure 6 FIG. 1 is a flow chart of a method for encoding a multi-frame video image provided by an embodiment of the present application, and the method specifically includes:

[0109] Step 601: The high-definition video live broadcast acquisition device 31 inputs the video.

[0110] The video is collected in real time by the high-definition live video collection device through the camera 318. As an example, the video can be a live video.

[0111] Step 602: The high-definition video live broadcast acquisition device 31 obtains a frame of image.

[0112] Step 603: The high-definition video live broadcast acquisition device 31 pre-configures the parameters of the Tile block, such as the parameter of the Tile block is N rows and M columns, which means that an image is divided into Tile blocks of N rows and M columns.

[0113] Step 604: NPU 313 runs the first-level coarse-tuning object detection model, inputs the entire image, and infers ROI-A.

[0114] Step 605: the high-definition video live broadcast acquisition device 31 divides ROI-A into multiple sub-ROIs according to the configured Tile blocks, and extracts the image of each sub-ROI.

[0115] Step 606: The high-definition live video acquisition device 31 adjusts the image of each sub-ROI to the same size and stacks them up as input to the secondary fine-tuning object detection model. The NPU 313 runs the secondary fine-tuning object detection model to infer the accurate ROI-B on the image of each sub-ROI.

[0116] Step 607: The high-definition video live broadcast acquisition device 31 performs intra-frame or inter-frame encoding on the current frame according to the encoding standard through the VPU 312, and encodes the image in the ROI-B in each Tile block in the image according to the encoding parameter 1, and encodes the image in the non-ROI area in each Tile block according to the encoding parameter 2.

[0117] The detailed description of the above steps 603-607 can refer to the steps 501-506 in the above embodiment, and the similarities are not repeated here. The difference is that when using the coded data for encoding, if the current frame is an I frame, intra-frame encoding is used, and if the current frame is a P or B frame, inter-frame encoding is used. More specifically, intra-frame encoding is used for Tile blocks belonging to I frames, and inter-frame encoding is used for Tile blocks belonging to P or B frames.

[0118] Step 608: The high-definition video live broadcast acquisition device 31 obtains the next frame of image and repeats steps 603-607.

[0119] Furthermore, in the video live broadcast scenario, after encoding the captured video, the high-definition video live broadcast acquisition device 31 can send the encoded video code stream to the cloud device through the network module 314. Subsequently, the cloud device can send the received video stream code to the high-definition video live broadcast terminal device 32. However, the high-definition video live broadcast terminal device 32 receives the video stream code through the network module 321, decodes the video stream code through the VPU 322, and then broadcasts the decoded video through the display unit 324 to complete the video live broadcast.

[0120] In the encoding method provided in the embodiment of the present application, while the encoding process of a frame of video image adopts Tile parallel encoding, the ROI area that focuses more on the object is used to encode the different parts on each block using appropriate encoding parameters. Therefore, it is possible to avoid high-quality encoding of excessive backgrounds in each frame image, avoid the waste of computing resources, improve the encoding efficiency of the video, and then reduce video transmission delay, bandwidth requirements, and transmission bit rate. In this way, the video live broadcast of the high-definition video live broadcast scene is smooth, clear, and has low latency.

[0121] The various embodiments of the mechanism disclosed in the present application can be implemented in hardware, software, firmware or a combination of these implementation methods. The embodiments of the present application can be implemented as a computer program or program code executed on a programmable system, which includes at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device and at least one output device.

[0122] Program code can be applied to input instructions to perform the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.

[0123] Program code can be implemented with high-level programming language or object-oriented programming language to communicate with the processing system. When necessary, program code can also be implemented with assembly language or machine language. In fact, the mechanism described in this application is not limited to the scope of any specific programming language. In either case, the language can be a compiled language or an interpreted language.

[0124] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, instructions may be distributed over a network or through other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine (e.g., computer) readable form, including, but not limited to, floppy disks, optical disks, optical disks, read-only memories (CD-ROMs), magneto-optical disks, read-only memories (ROMs), random access memories (RAMs), erasable programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs), magnetic or optical cards, flash memory, or a tangible machine-readable memory for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in electrical, optical, acoustic, or other forms of propagation signals. Therefore, machine-readable media include any type of machine-readable media suitable for storing or transmitting electronic instructions or information in a machine (e.g., computer) readable form.

[0125] In the accompanying drawings, some structural or method features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be required. Instead, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. In addition, the inclusion of structural or method features in a particular figure does not mean that such features are required in all embodiments, and in some embodiments, these features may not be included or may be combined with other features.

[0126] It should be noted that the units / modules mentioned in the various device embodiments of the present application are all logical units / modules. Physically, a logical unit / module can be a physical unit / module, or a part of a physical unit / module, or can be implemented as a combination of multiple physical units / modules. The physical implementation method of these logical units / modules themselves is not the most important. The combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed by the present application. In addition, in order to highlight the innovative part of the present application, the above-mentioned device embodiments of the present application do not introduce units / modules that are not closely related to solving the technical problems proposed by the present application, which does not mean that there are no other units / modules in the above-mentioned device embodiments.

[0127] It should be noted that in the examples and description of this patent, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one" do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0128] Although the present application has been illustrated and described with reference to certain preferred embodiments thereof, it will be apparent to those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present application.

Claims

1. A coding method, characterized in that: include: Get the image to be processed; Dividing the image to be processed into multiple blocks according to the Tile block encoding mechanism; Detecting the image to be processed to detect an original area containing a target object in the image to be processed; Dividing the original area into the multiple blocks to obtain multiple sub-areas; Detecting the image in the sub-region in each of the blocks, and detecting the target region in each of the blocks that contains the target object; The image in the target area in each of the blocks is encoded according to the first encoding parameter, and the image in the area other than the target area in each of the block images is encoded according to the second encoding parameter, so as to encode the multiple blocks in parallel, wherein the image encoding quality corresponding to the first encoding parameter is higher than the image encoding quality corresponding to the second encoding parameter.

2. The method according to claim 1, characterized in that The detecting the image to be processed to detect the original area containing the target object in the image to be processed includes: The image to be processed is adjusted to a first image size and input into a first object detection model, and the first object detection model outputs the original area in the image to be processed; The detecting of the image in the sub-region in each of the blocks to detect the target region in each of the blocks containing the target object comprises: The image in the sub-region in each of the blocks is adjusted to a second image size and input into a second object detection model, and the second object detection model outputs the target region in each of the blocks.

3. The method according to claim 2, characterized in that The image to be processed is any frame of image in the video to be processed; When the image to be processed is an I frame in a video to be processed, each block in the image to be processed is encoded based on intra-frame coding according to the first encoding parameter and the second encoding parameter; When the image to be processed is a P frame or a B frame in the image to be processed, each block in the image to be processed is encoded based on inter-frame coding according to the first encoding parameter and the second encoding parameter.

4. The method according to claim 3, characterized in that The method further comprises: When the to-be-processed video is encoded into a video code stream based on the first encoding parameter and the second encoding parameter, sending the video code stream to a cloud device; The cloud device is used to send the video code stream to a receiving device, and the receiving device is used to decode the video code stream and play the video to be processed.

5. An electronic device, characterized in that: include: An acquisition module, used for acquiring an image to be processed; A division module, used for dividing the image to be processed acquired by the acquisition module into a plurality of blocks according to a Tile block coding mechanism; A detection module, configured to detect the image to be processed acquired by the acquisition module, detect an original region containing a target object in the image to be processed; and divide the original region into the multiple blocks to obtain multiple sub-regions; Detecting the image in the sub-region in each of the blocks, and detecting the target region in each of the blocks that contains the target object; A coding module is used to encode the image in the target area in each of the blocks detected by the detection module according to a first coding parameter, and to encode the image in the area other than the target area in each of the block images according to a second coding parameter, so as to encode the multiple blocks in parallel, wherein the image coding quality corresponding to the first coding parameter is higher than the image coding quality corresponding to the second coding parameter.

6. The device according to claim 5, characterized in that The detection module is specifically used to adjust the image to be processed to a first image size and input the image into a first object detection model, and the first object detection model outputs the original area in the image to be processed; And, the image in the sub-region in each of the blocks is adjusted to a second image size and input into a second object detection model, and the second object detection model outputs the target region in each of the blocks.

7. The device according to claim 6, characterized in that The image to be processed is any frame of image in the video to be processed; When the image to be processed is an I frame in a video to be processed, each block in the image to be processed is encoded based on intra-frame coding according to the first encoding parameter and the second encoding parameter; When the image to be processed is a P frame or a B frame in the image to be processed, each block in the image to be processed is encoded based on inter-frame coding according to the first encoding parameter and the second encoding parameter.

8. The device according to claim 7, characterized in that The device also includes: A sending module, configured to, when encoding the to-be-processed video into a video stream based on the first encoding parameter and the second encoding parameter, send the video stream to a cloud device; The cloud device is used to send the video code stream to a receiving device, and the receiving device is used to decode the video code stream and play the video to be processed, and the video to be processed is a live video.

9. A readable medium, characterized in that The readable medium stores instructions, which, when executed on an electronic device, enable the electronic device to execute the encoding method according to any one of claims 1 to 4.

10. An electronic device, characterized in that: The device comprises: a memory for storing instructions executed by one or more processors of the electronic device, and a processor, which is one of the processors of the electronic device, for executing the encoding method according to any one of claims 1 to 4.

Images