Video coding method, decoding method and related device

Abstract

A video coding method is applied to a video coding end. In the method, when an image in a video is coded, obtained ROI information of the image is packaged into coded data of the image at the same time, so that the image and the ROI information of the image are transmitted to a video decoding end at the same time, and the video decoding end can further process the video based on the ROI information of the image; the video decoding end is prevented from recognizing the ROI again based on the decoded image, and effective utilization of the ROI information of the image is ensured.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of video processing technology, and in particular to a video encoding method, a decoding method and related devices. Background Art

[0002] With the popularization of the Internet, video transmission is becoming more and more common in various fields of society. In some specific scenarios (such as security scenarios and multimedia scenarios), video transmission has become an indispensable role.

[0003] Generally, in the process of video transmission, in order to reduce the amount of video data that needs to be transmitted, the video is usually encoded to achieve compression of the video data. In the related art, in order to further reduce the amount of compressed video data, the region of interest (ROI) encoding technology is used to encode the video. The ROI encoding technology specifically identifies the ROI of each image in the video, thereby assigning different compression parameters to the ROI and non-ROI in the image, and further reducing the quality of the non-ROI in the image while ensuring the quality of the ROI in the image, thereby achieving further reduction of the amount of compressed video data.

[0004] However, in the related art, after the ROI in the image is identified, different compression encoding is performed on different areas of the image based on the ROI in the image only, and the ROI of the image is not further effectively utilized, resulting in a waste of the ROI information of the image. Summary of the invention

[0005] The present application provides a video encoding method, which can ensure the effective use of the ROI information of the image by encapsulating the ROI information of the image into a bit stream.

[0006] In a first aspect, the present application provides a video encoding method, which is applied to a video encoding end. The method includes: the video encoding end obtains ROI information, and the ROI information is used to indicate the ROI in a first image, and the first image is an image in the video to be encoded. The ROI is a part of the image area in the first image, and the content included in the ROI is the content that the user is interested in. For example, when the acquisition scene of the first image is a security scene, the ROI generally refers to the area in the image that includes pedestrians, motor vehicles, and non-motor vehicles.

[0007] In the process of encoding the first image, the video encoding end encapsulates the ROI information in the encoded data of the first image. That is, the video encoding end can regard the ROI information of the first image as part of the content of the first image, and also encode the ROI information of the first image, thereby achieving encapsulation of the ROI information of the first image in the encoded data of the first image.

[0008] Finally, the video encoding end sends the encoded data of the first image so that the video decoding end can receive the encoded data of the first image.

[0009] In this solution, when encoding an image in a video, the ROI information of the acquired image is simultaneously encapsulated into the encoded data of the image, thereby realizing the simultaneous transmission of the image and the ROI information of the image to the video decoding end, so that the video decoding end can further process the video based on the ROI information of the image, avoiding the video decoding end from re-identifying the ROI based on the decoded image, and ensuring the effective use of the ROI information of the image.

[0010] In a possible implementation, the ROI information is encapsulated in any of the following fields in the coded data: Supplement Enhancement Information (SEI) field, Instantaneous Decoding Refresh (IDR) field, Sequence Parameter Set (SPS) field, Picture Parameter Set (PPS) field, or Picture Coded Slice field. For example, the ROI information may be encapsulated in the original flag bit of the above fields, or in the flag bit of the above fields extension.

[0011] In this solution, by encapsulating the ROI information of the image in the original field of the coded data, it is possible to achieve the encapsulation of the ROI information of the image while minimizing the changes to the existing technology, thereby improving the feasibility of the solution.

[0012] In a possible implementation, the ROI information encapsulated in the encoded data includes a ROI mask map, the ROI mask map includes a plurality of pixels, and the value of each pixel in the plurality of pixels is used to indicate whether each pixel belongs to the ROI.

[0013] That is, the ROI information of the first image is expressed in the form of a ROI mask map, that is, the value of each pixel in the ROI mask map indicates whether each pixel belongs to the ROI of the first image. In this way, by searching for pixels with specific values ​​in the ROI mask map, the position of the ROI of the first image can be determined. For example, in the ROI mask map, the value of a pixel located outside the ROI can be 0, and the value of a pixel located in the ROI can be a non-zero value.

[0014] In this solution, by representing the ROI information in the form of a ROI mask map, it is possible to clearly indicate whether each pixel in the image belongs to the ROI, and the ROI mask map can be used as a part of the image information. The ROI mask map is compressed and encoded in the same manner as the compressed encoded image, thereby minimizing the changes to the existing technology and ensuring that valid ROI information is carried in the encoded data of the image.

[0015] In a possible implementation, the value of the target pixel in the ROI mask is also used to indicate the category of the ROI to which the target pixel belongs, and the target pixel is the pixel in the ROI mask that belongs to the ROI. That is, for the pixels located in the ROI, the values ​​of these pixels can indicate not only whether the pixels are located in the ROI, but also the category of the ROI where these pixels are located.

[0016] In this scheme, by assigning different values ​​to different pixels in the ROI mask image, the category of the ROI to which the pixel belongs can be further distinguished based on the pixel value, thereby providing ROI information with richer semantic information and ensuring that the video decoding end can effectively use the ROI information to perform subsequent video processing tasks.

[0017] In a possible implementation, the ROI information is also used to indicate the total number of ROI categories in the first image, and the value of the target pixel in the ROI mask image is related to the category of the ROI to which the target pixel belongs and the total number of ROI categories in the first image. The ROI information may directly indicate the total number of ROI categories in the first image; for example, in a traffic scene, the ROI categories include pedestrian areas, motor vehicle areas, and non-motor vehicle areas, and the ROI information may indicate that the total number of ROI categories in the first image is 3.

[0018] The ROI information may also directly indicate the total number of categories of the region in the first image. Since the first image necessarily includes non-ROIs, the total number of categories of the region in the first image is actually the total number of categories of the ROI in the first image plus 1. The total number of categories of the ROI in the first image can also be determined based on the total number of categories of the region in the first image indicated in the ROI information, that is, the ROI information indirectly indicates the total number of categories of the ROI in the first image.

[0019] Furthermore, in order to facilitate accurate determination of the category of the region to which each pixel belongs during the decoding process, the actual value of each pixel can be determined based on the category of the ROI to which the pixel belongs and the total number of categories of the ROI in the first image during the encoding process of the ROI mask map, that is, the value of the target pixel in the ROI mask map is related to the category of the ROI to which the target pixel belongs and the total number of categories of the ROI in the first image. For example, when the value range of the pixel in the ROI mask map is 0-255, and the ROI category to which the pixel belongs includes a pedestrian area, a motor vehicle area, and a non-motor vehicle area, it can be determined that there are 4 area categories corresponding to the pixel in the ROI mask map (i.e., non-ROI, pedestrian area, motor vehicle area, and non-motor vehicle area), so the value range (0-255) can be divided into four intervals, and a certain value in each interval (e.g., the center value of the interval) is used as the value corresponding to each category. Specifically, the value range of 0-255 can be divided into four intervals: 0-63, 64-127, 128-191, and 192-255; and, when the value of the pixel point is 32 (i.e., the center value of the interval 0-63), it represents that the region category to which the pixel point belongs is non-ROI; when the value of the pixel point is 96 (i.e., the center value of the interval 64-127), it represents that the region category to which the pixel point belongs is the pedestrian region; when the value of the pixel point is 160 (i.e., the center value of the interval 128-191), it represents that the region category to which the pixel point belongs is the motor vehicle region; when the value of the pixel point is 224 (i.e., the center value of the interval 192-255), it represents that the region category to which the pixel point belongs is the non-motor vehicle region.

[0020] In this scheme, by determining the actual value of the pixel in the ROI mask image based on the total number of ROI categories, the values ​​of the pixels belonging to different regions can be staggered as much as possible (that is, the difference between the values ​​is as large as possible), thereby ensuring that the quantization loss caused by the encoding process will not affect the determination of the region category to which each pixel belongs during the decoding process, thereby ensuring the feasibility of the scheme.

[0021] In a possible implementation, the ROI mask image is obtained after downsampling, and the ROI mask image is smaller than the first image. For example, the ROI mask image encapsulated in the encoded data may be obtained after downsampling the original mask image, and the size of the original mask image is the same as that of the first image, and each pixel in the original mask image is used to indicate the region to which the corresponding pixel in the first image belongs.

[0022] In this solution, by encapsulating the ROI mask image obtained after downsampling into the encoded data, the data volume of the ROI information can be greatly reduced while ensuring the accuracy of the ROI information as much as possible, reducing the data volume of the encoded data and the computing resources required for the encoding process.

[0023] In a possible implementation, the ROI information encapsulated in the encoded data includes one or more point sets, each point set includes multiple points, and the closed area surrounded by the multiple points in each point set belongs to the ROI in the first image. In other words, a point set is used to represent a closed area, and the closed area represented by each point set belongs to the ROI in the image, that is, the area of ​​the image belonging to the ROI is represented by multiple points located on the edge of the area. Generally speaking, the number of point sets in the ROI information corresponds to the number of ROIs included in the image. The more ROIs separated in the image, the more point sets there are.

[0024] In this scheme, by using a set of points that can enclose a closed area to represent the ROI in the image, it is possible to represent a larger ROI in the image with a small number of points, avoiding the use of all pixels in the ROI to represent the ROI, and effectively representing the ROI in the image in the form of low data volume, thereby ensuring the feasibility of the scheme.

[0025] In a possible implementation, the ROI information encapsulated in the encoded data is also used to indicate the category of the ROI to which the closed area surrounded by each point set belongs. Specifically, the ROI information can indicate the category of the ROI to which the closed area surrounded by each point set belongs through a specific flag bit. For example, the flag bit can take values ​​such as 1, 2, and 3 to indicate that the category of the ROI to which the closed area surrounded by each point set belongs is a pedestrian area, a motor vehicle area, or a non-motor vehicle area.

[0026] In a possible implementation, the ROI information is obtained by identifying the first image through an artificial intelligence model.

[0027] In a possible implementation manner, the acquisition scene of the first image is a security scene.

[0028] A second aspect of the present application provides a video decoding method, which is applied to a video decoding end, and the method includes: the video decoding end receives encoded data of a first image, the encoded data encapsulated with ROI information, the ROI information is used to indicate the ROI in the first image, and the first image is an image in the video; the video decoding end decodes the encoded data to obtain the first image and the ROI information.

[0029] In a possible implementation, the ROI information is used to perform any one of the following tasks on the video: video compression, video tagging, live album, and video condensation.

[0030] In a possible implementation manner, the ROI information is encapsulated in any one of the following fields in the coded data: a SEI field, an IDR field, an SPS field, a PPS field, or an image coding slice field.

[0031] In a possible implementation, the ROI information encapsulated in the encoded data includes a ROI mask map, the ROI mask map includes a plurality of pixels, and the value of each pixel in the plurality of pixels is used to indicate whether each pixel belongs to the ROI.

[0032] In a possible implementation, the value of the target pixel in the ROI mask image is also used to indicate the category of the ROI to which the target pixel belongs. The target pixel is a pixel in the ROI mask image that belongs to the ROI.

[0033] In a possible implementation, the ROI information is also used to indicate the total number of categories of ROI in the first image. In the process of decoding the encoded data, the ROI mask map in the encoded data can be decoded first to obtain the value of each pixel in the ROI mask map. Due to the quantization loss of the encoding process, the value of the pixel obtained by decoding may be different from the value corresponding to the category to which the pixel actually belongs. Therefore, the category of the area to which each pixel belongs can be determined based on the target value range of the value of each pixel in multiple value ranges. Among them, the multiple value ranges are determined based on the total number of categories of ROI in the first image.

[0034] For example, when the value range of the pixel in the ROI mask is 0-255, and the ROI category to which the pixel belongs includes pedestrian area, motor vehicle area and non-motor vehicle area, it can be determined that there are four area categories corresponding to the pixel in the ROI mask (i.e., non-ROI, pedestrian area, motor vehicle area and non-motor vehicle area), so the total value range (0-255) can be divided into four small value ranges, so as to determine the area category to which the pixel belongs based on the value range in which the actual value of the pixel is located. Specifically, the total value range 0-255 can be divided into four small value ranges of 0-63, 64-127, 128-191 and 192-255. Furthermore, if the value of the pixel point is in the value range of 0-63, it means that the region category to which the pixel point belongs is non-ROI; if the value of the pixel point is in the value range, it means that the region category to which the pixel point belongs is the pedestrian region; if the value of the pixel point is in the value range, it means that the region category to which the pixel point belongs is the motor vehicle region; if the value of the pixel point is in the value range, it means that the region category to which the pixel point belongs is the non-motor vehicle region.

[0035] In this scheme, multiple value ranges are determined based on the total number of ROI categories, and the region category to which the pixel belongs is determined based on the value range of the pixel value obtained by decoding, thereby ensuring that the quantization loss caused by the encoding process will not affect the determination of the region category to which each pixel belongs during the decoding process, thereby ensuring the feasibility of the scheme.

[0036] In a possible implementation manner, the ROI mask image is obtained after downsampling, and the ROI mask image is smaller than the first image.

[0037] In a possible implementation, the ROI information encapsulated in the encoded data includes one or more point sets, each point set includes multiple points, and a closed area surrounded by the multiple points in each point set belongs to the ROI in the first image.

[0038] In a possible implementation, the ROI information encapsulated in the encoded data is also used to indicate the category of the ROI to which the closed area surrounded by each point set belongs.

[0039] In a possible implementation, the ROI information is obtained by identifying the first image through an artificial intelligence model.

[0040] In a possible implementation manner, the acquisition scene of the first image is a security scene.

[0041] The third aspect of the present application provides a video encoding device, including: an acquisition module, used to obtain region of interest ROI information, the ROI information is used to indicate the ROI in a first image, and the first image is an image in the video to be encoded; an encoding module, used to encapsulate the ROI information in the encoding data of the first image during the encoding of the first image; and a sending module, used to send the encoding data of the first image.

[0042] In a possible implementation, the ROI information is encapsulated in any one of the following fields in the coded data: a supplemental enhancement information SEI field, an instantaneous decoding refresh IDR field, a sequence parameter set SPS field, a picture parameter set PPS field, or a picture coded slice field.

[0043] In a possible implementation, the ROI information encapsulated in the encoded data includes a ROI mask map, the ROI mask map includes a plurality of pixels, and the value of each pixel in the plurality of pixels is used to indicate whether each pixel belongs to the ROI.

[0044] In a possible implementation, the value of the target pixel in the ROI mask image is also used to indicate the category of the ROI to which the target pixel belongs. The target pixel is a pixel in the ROI mask image that belongs to the ROI.

[0045] In a possible implementation, the ROI information is also used to indicate the total number of ROI categories in the first image, and the value of the target pixel in the ROI mask image is related to the category of the ROI to which the target pixel belongs and the total number of ROI categories in the first image.

[0046] In a possible implementation manner, the ROI mask image is obtained after downsampling, and the ROI mask image is smaller than the first image.

[0047] In a possible implementation, the ROI information encapsulated in the encoded data includes one or more point sets, each point set includes multiple points, and a closed area surrounded by the multiple points in each point set belongs to the ROI in the first image.

[0048] In a possible implementation, the ROI information encapsulated in the encoded data is also used to indicate the category of the ROI to which the closed area surrounded by each point set belongs.

[0049] In a possible implementation, the ROI information is obtained by identifying the first image through an artificial intelligence model.

[0050] In a possible implementation manner, the acquisition scene of the first image is a security scene.

[0051] In a fourth aspect, the present application provides a video decoding device, comprising: a receiving module, used to receive encoded data of a first image, wherein the encoded data encapsulates ROI information, and the ROI information is used to indicate the ROI in the first image, where the first image is an image in the video; and a decoding module, used to decode the encoded data to obtain the first image and the ROI information.

[0052] In a possible implementation, the ROI information is used to perform any one of the following tasks on the video: video compression, video tagging, live album, and video condensation.

[0053] In a possible implementation manner, the ROI information is encapsulated in any one of the following fields in the coded data: a SEI field, an IDR field, an SPS field, a PPS field, or an image coding slice field.

[0054] In a possible implementation, the ROI information encapsulated in the encoded data includes a ROI mask map, the ROI mask map includes a plurality of pixels, and the value of each pixel in the plurality of pixels is used to indicate whether each pixel belongs to the ROI.

[0055] In a possible implementation, the value of the target pixel in the ROI mask image is also used to indicate the category of the ROI to which the target pixel belongs. The target pixel is a pixel in the ROI mask image that belongs to the ROI.

[0056] In a possible implementation manner, the ROI information is further used to indicate the total number of categories of ROIs in the first image;

[0057] The decoding module is also used to: decode the ROI mask image in the encoded data to obtain the value of each pixel in the ROI mask image; determine the category of the area to which each pixel belongs based on the target value range of the value of each pixel in multiple value ranges; wherein the multiple value ranges are determined based on the total number of categories of the ROI in the first image.

[0058] In a possible implementation manner, the ROI mask image is obtained after downsampling, and the ROI mask image is smaller than the first image.

[0059] In a possible implementation, the ROI information encapsulated in the encoded data includes one or more point sets, each point set includes multiple points, and a closed area surrounded by the multiple points in each point set belongs to the ROI in the first image.

[0060] In a possible implementation, the ROI information encapsulated in the encoded data is also used to indicate the category of the ROI to which the closed area surrounded by each point set belongs.

[0061] In a possible implementation, the ROI information is obtained by identifying the first image through an artificial intelligence model.

[0062] In a possible implementation manner, the acquisition scene of the first image is a security scene.

[0063] The fifth aspect of the present application provides a video encoding device, including: a processor and a memory; the memory is used to store computer instructions, and when the processor executes the instruction, the video encoding device executes the method of any implementation method of the above-mentioned first aspect.

[0064] The sixth aspect of the present application provides a video decoding device, including: a processor and a memory; the memory is used to store computer instructions, and when the processor executes the instruction, the video decoding device executes the method of any implementation method of the above-mentioned second aspect.

[0065] In a seventh aspect, the present application provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on an electronic device, the electronic device can execute any of the methods described above.

[0066] An eighth aspect of the present application provides a computer program product comprising instructions, which, when executed on an electronic device, enables the electronic device to execute any of the methods described above.

[0067] In the ninth aspect of the present application, a chip is provided, which includes a processor and a communication interface, the communication interface is used to communicate with modules outside the chip shown, and the processor is used to run computer programs or instructions so that a device equipped with the chip can execute any of the methods described above.

[0068] Among them, the technical effects brought about by any design method in the second to ninth aspects can refer to the technical effects brought about by different implementation methods in the above-mentioned first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 A schematic diagram of a data structure of a code stream provided in an embodiment of the present application;

[0070] Figure 2 A schematic diagram of a security scenario provided in an embodiment of the present application;

[0071] Figure 3 A schematic diagram of a video encoding and decoding method provided in an embodiment of the present application;

[0072] Figure 4 A schematic diagram of performing ROI recognition on an image provided in an embodiment of the present application;

[0073] Figure 5 A schematic diagram of encoding data of an image provided in an embodiment of the present application;

[0074] Figure 6 A schematic diagram of a processing flow of a ROI mask image provided in an embodiment of the present application;

[0075] Figure 7 A schematic diagram of an ROI mask image provided in an embodiment of the present application being transmitted along with an I frame and a P frame;

[0076] Figure 8 A schematic diagram of an embodiment of the present application providing a method of shifting the pixel values ​​in the ROI mask image left by 8-k bits;

[0077] Fig. 9 A schematic diagram of a packaged ROI mask image provided in an embodiment of the present application;

[0078] Fig.10 A schematic diagram of representing the ROI of an image by a set of points provided in an embodiment of the present application;

[0079] Fig.11 A schematic diagram of a processing flow of a point set in ROI information provided in an embodiment of the present application;

[0080] Fig.12 A schematic diagram of a packaging point structure data provided by an embodiment of the present application;

[0081] Fig.13 A schematic diagram of the structure of a video encoding device provided in an embodiment of the present application;

[0082] Fig.14 A schematic diagram of the structure of a video decoding device provided in an embodiment of the present application;

[0083] Fig.15 A schematic diagram of the structure of a chip according to an embodiment of the present application. DETAILED DESCRIPTION

[0084] In order to make the purpose, technical solutions and advantages of the present application clearer, the embodiments of the present application are described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only embodiments of a part of the present application, rather than all embodiments. It is known to those of ordinary skill in the art that with the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0085] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the descriptions used in this way can be interchanged where appropriate, so that the embodiments can be implemented in a sequence other than that illustrated or described in the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or modules is not necessarily limited to those steps or modules that are clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products or devices. The naming or numbering of the steps that appear in the present application does not mean that the steps in the method flow must be executed in the time / logical sequence indicated by the naming or numbering. The process steps that have been named or numbered can change the execution order according to the technical purpose to be achieved, as long as the same or similar technical effects can be achieved.

[0086] The division of units in this application is a logical division. There may be other division methods when it is implemented in actual applications. For example, multiple units can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection between units can be electrical or other similar forms, which are not limited in this application. In addition, the units or sub-units described as separate components may or may not be physically separated, may or may not be physical units, or may be distributed in multiple circuit units, and some or all of the units may be selected according to actual needs to achieve the purpose of the present application.

[0087] To facilitate understanding, some technical terms involved in the embodiments of the present application are first introduced below.

[0088] (1) Video Coding

[0089] Video encoding refers to the process of converting an original video format file into another video format file through compression technology.

[0090] Specifically, a video is a continuous sequence of images, which is essentially composed of continuous frames, one frame being one image. Due to the persistence of vision of the human eye, when an image sequence is played at a certain rate, what people see is a video with continuous action. Since the similarity between continuous frames is extremely high, in order to facilitate video transmission, the original video is usually encoded and compressed to remove redundancy in the spatial and temporal dimensions.

[0091] At present, commonly used video coding standards are H.264 and H.265. Taking the H.264 coding standard as an example, the video coding process includes the following steps: image segmentation, intra-frame prediction, inter-frame prediction, transformation, quantization and entropy coding.

[0092] In addition, when encoding each image in a video, each image can be divided into multiple image coding slices, and each image coding slice can be divided into multiple macroblocks (i.e., multiple image blocks). Among them, macroblock is the smallest processing unit of video coding and carries specific information of the video.

[0093] (2) Image segmentation

[0094] Image segmentation refers to dividing a large image in the original video into multiple small image blocks.

[0095] (3) Intra-frame prediction

[0096] Intra-frame prediction refers to using the boundary pixels of adjacent image blocks as reference pixels in the current frame to predict the pixels of the current image block, with the aim of removing spatial redundancy of the video signal.

[0097] For a single-frame image, the grayscale change of the current pixel of the image is very small compared with the adjacent pixels at the top, bottom, left, and right positions in the frame, and the correlation is extremely strong. There is spatial redundancy in the image, and intra-frame prediction can reduce the spatial redundancy in the above image.

[0098] (4) Inter-frame prediction

[0099] Inter-frame prediction uses the correlation between video image frames, that is, temporal correlation, to achieve the purpose of image compression. It is widely used in compression coding of ordinary television, conference television, video telephone, and high-definition television.

[0100] Specifically, inter-frame prediction uses adjacent encoded images to find the best matching block for the coding block in the current image, and uses this best matching block as the prediction value of the current coding block. The prediction value is then subtracted from the original pixel value of the current coding block to obtain the residual value of the current coding block. Subsequent transformation, quantization and other operations are all based on the residual value.

[0101] In the process of inter-frame prediction, the best matching block is found for the coding block in the current image by motion estimation, and a motion vector (MV) is calculated. The motion vector is used to represent the relative displacement between the current coding block and the best matching block in the reference image.

[0102] In general, since the video is composed of multiple frames and contains multiple frames per second, the pixels of the current image are very similar to the pixels at corresponding positions in the previous and next frames, and are highly correlated, and there is temporal redundancy. The use of inter-frame prediction can reduce the temporal redundancy in the above images.

[0103] (5) Transformation

[0104] Transformation refers to converting an image described in the form of pixels in the spatial domain to a transform domain (usually the frequency domain) and expressing it in the form of transform coefficients. Generally speaking, most areas in an image are flat areas and areas where the content changes slowly, that is, most of the content in the image is low-frequency information (such as the overall contour information of the image), and the high-frequency information in the image (such as the edge or detail information of the image) is often relatively small, so appropriate transformation can convert the dispersed distribution of image energy in the spatial domain into a relatively concentrated distribution in the transform domain, so as to achieve the purpose of removing redundancy. In general, transformation refers to the conversion of an image from the time domain to the frequency domain, thereby removing the correlation between adjacent data, that is, removing spatial redundancy.

[0105] Specifically, the transformation does not directly encode the spatial image signal, but first maps the spatial image signal to another orthogonal vector space (transform domain or frequency domain), generates a batch of transform coefficients, and then encodes these transform coefficients. Transformation is actually an indirect coding method. The key problem is that when describing in the time domain or spatial domain, the data has a large correlation and data redundancy. After the transformation is described in the transform domain, the data correlation is greatly reduced, the data redundancy is reduced, the parameters are independent, and the amount of data is small. In this way, after the video data is transformed and then quantized, the encoding can obtain a larger compression ratio. Typical transformation methods include discrete cosine transform (DCT), discrete Fourier transform (DFT), etc.

[0106] (6) Quantification

[0107] Quantization refers to the mapping of transform coefficients to a finite number of quantization coefficient values, which can realize the mapping of a data set containing a large amount of data to a data set containing a small amount of data. Simply put, quantization is to reduce the amount of encoded data by representing fine data with coarser data, or by removing information that is not sensitive to the human eye.

[0108] In the video encoding process, the quantization process is the only lossy process, which mainly determines the compression rate of the entire video. The higher the compression rate of the video, the worse the video quality and the lower the bit rate; conversely, the lower the video compression rate, the better the video quality and the higher the bit rate. Specifically, quantization is the process of mapping the continuous value of the signal into multiple discrete amplitudes, realizing a many-to-one mapping of signal values. In the transformation step, after the residual data is transformed, the transform coefficient has a larger value range. Quantization can effectively reduce the value range of the signal, thereby obtaining a better compression effect.

[0109] Generally speaking, the quantization parameter (QP) in the quantization process corresponds to the serial number of the quantization step. The size of the quantization parameter can reflect the compression of spatial details. The smaller the quantization parameter, the finer the quantization, the more details retained in the image, the higher the image quality, and the larger the storage space occupied by the image; the larger the quantization parameter, the coarser the quantization, the more details lost in the image, the lower the image quality, and the smaller the storage space occupied by the image.

[0110] (7) Entropy Coding

[0111] Entropy coding is a type of coding that uses the statistical characteristics of the source to perform bit rate compression. The basic idea of ​​entropy coding is to make the code words before and after as random as possible, reduce the correlation between the before and after, and get closer to the Shannon entropy of the source. In general, entropy coding has the function of eliminating statistical redundancy between data. It is usually used as the last step at the encoding end to write syntax elements into the output bit stream.

[0112] (8)YUV

[0113] YUV is a color encoding method, where Y represents brightness, and U and V represent chrominance. Compared with RGB, YUV is more suitable for encoding and decoding.

[0114] (9) ROI

[0115] In the field of image processing, ROI is an image region selected from an image, and this image region is the focus of image analysis. In different scenarios, the definition of ROI may be different, but ROI is essentially the area that people focus on when performing image analysis. For example, in security scenarios, ROI usually refers to the area in the image that includes pedestrians, motor vehicles, and non-motor vehicles; for another example, in autonomous driving scenarios, ROI usually refers to the area in the image that includes pedestrians, motor vehicles, non-motor vehicles, traffic lights, obstacles and other objects.

[0116] (10) Keyframe

[0117] In video coding, a key frame refers to an image frame whose content needs to be fully encoded. It is an independent frame that carries all the information and can be decoded independently without referring to other image frames. When decoding a key frame, you only need to decode the key frame itself, because the encoded data of the key frame already includes all the content in the key frame. Generally, a key frame is also called an I frame.

[0118] (11) Non-keyframe

[0119] In video coding, non-key frames refer to image frames other than key frames. Generally speaking, when encoding non-key frames, the difference between non-key frames and other frames is actually encoded. In this way, when decoding non-key frames, it is necessary to combine the picture content of other frames and the decoded content of non-key frames to generate the actual picture of non-key frames.

[0120] A common non-key frame is a forward predicted frame (P frame). When encoding a P frame, it is usually encoded based on the difference between the current P frame and the previous frame, thereby reducing the amount of encoded data of the P frame.

[0121] (12) Bitstream

[0122] A bitstream is a coded data stream obtained by encoding the images in a video. In each video coding standard, the data structure in the bitstream is defined to indicate the content information carried by each part of the data in the bitstream.

[0123] Taking the H.264 video coding standard as an example, H.264 introduces a layered structure, which divides the image compression system into a video coding layer (VCL) and a network abstraction layer (NAL), thereby separating compression coding from network transmission. Among them, VCL data is the compressed and encoded video data sequence, and VCL data needs to be encapsulated into a network abstraction layer unit (NALU) before it can be transmitted. Therefore, you can refer to Figure 1 , Figure 1 A schematic diagram of a data structure of a code stream provided in an embodiment of the present application. Figure 1 As shown, in H.264, the code stream actually includes a series of NALUs, and each NALU includes a NALU header and a raw byte sequence payload (RBSP). Among them, the NALU header can be used to indicate the type of the current NALU, that is, the type of data carried by the RBSP in the current NALU.

[0124] Exemplarily, the type field in the NALU header can indicate the type of the current NALU through different values, as shown in Table 1.

[0125]

[0126]

[0127] (13) Downsampling

[0128] Downsampling, also known as downsampling, is a method of reducing an image to make it fit the required size, that is, generating a thumbnail of the corresponding image.

[0129] (14) Upsampling

[0130] Upsampling is a method of enlarging an image. Usually, upsampling is to insert some new pixels on the basis of the pixels of the original image through interpolation, so as to increase the pixels of the image and enlarge the image.

[0131] In the current ROI encoding technology, usually before encoding the image in the video, an AI model is used to identify the ROI of the image, so as to identify the ROI and non-ROI in the image. In this way, in the process of encoding the image, different compression parameters can be assigned to the ROI and non-ROI in the image, and the quality of the non-ROI in the image can be further reduced while ensuring the quality of the ROI in the image, thereby further reducing the amount of data after compression of the video data.

[0132] However, in the related art, after recognizing the ROI in the image, different compression encoding is performed on different areas of the image based on the ROI in the image, and the ROI of the image is not further effectively utilized, resulting in a waste of the ROI information of the image. For example, when the upper layer application at the video decoding end needs to use the ROI information of the image in the video to further process the video, the upper layer application at the video decoding end often needs to re-perform ROI recognition on the decoded image, resulting in repeated execution of ROI recognition.

[0133] In view of this, the present embodiment provides a video encoding method, which, when encoding an image in a video, encapsulates the acquired ROI information of the image into the encoded data of the image at the same time, thereby realizing the simultaneous transmission of the image and the ROI information of the image to the video decoding end, so that the video decoding end can further process the video based on the ROI information of the image, avoiding the video decoding end from re-identifying the ROI based on the decoded image, and ensuring the effective use of the ROI information of the image.

[0134] Exemplarily, the scenario in which the video encoding method provided in this embodiment is applied may be a security scenario.

[0135] See also Figure 2 , Figure 2 A schematic diagram of a security scenario provided in an embodiment of the present application. Figure 2As shown, in the security scenario, the intelligent security system can usually be composed of an acquisition module, a storage module and an analysis module. For example, the acquisition module may include a security camera, an internet protocol camera (IPC) or a software defined camera (SDC), etc., the storage module may include a video cloud node (VCN), etc., and the analysis module may include a video content management system (VCM), etc. The acquisition module at the front end acquires the video, compresses and encodes the video, and transmits the encoded video data to the storage module at the back end through the network. The storage module stores the acquired video data. The analysis module extracts the video data stored in the storage module, performs intelligent identification and analysis on the video data, and obtains the analysis results.

[0136] The video encoding method provided in this embodiment can be applied to the front-end acquisition module in the security scene, for example, to a security camera with video encoding capability. In this way, after acquiring the image, the security camera can determine the ROI in the image based on the method provided in this embodiment, that is, the area where pedestrians, motor vehicles and non-motor vehicles are located, and then assign different compression encoding parameters to the ROI and non-ROI in the image during the video encoding process, and add the ROI information of the image during the video encoding process, so that the encoded data of the image includes both the image and the ROI of the image.

[0137] The video decoding method provided in this embodiment can be applied to a storage module or an analysis module in a security scenario. By decoding the encoded data received from the acquisition module, the image in the video and the ROI of the image are obtained, and the decoded video can be further processed accordingly based on the ROI of the image.

[0138] The above introduces the application scenarios of a video encoding method and a video decoding method (hereinafter referred to as the video encoding and decoding method) provided by this embodiment. The following will introduce the execution process of the video encoding and decoding method in detail.

[0139] See also Figure 3 , Figure 3 A schematic diagram of a video encoding and decoding method provided in an embodiment of the present application. Figure 3 As shown, the video encoding method is applied to a video encoding process, and specifically includes the following steps 301-303.

[0140] Step 301: The video encoding end obtains ROI information, where the ROI information is used to indicate the ROI in the first image.

[0141] In this embodiment, the video encoding end is used to encode continuous images in the video to output a code stream that can be transmitted on the network. Before encoding the first image in the video, the video encoding end can obtain ROI information of the first image, and the ROI information can indicate the ROI in the first image. Among them, the ROI is a part of the image area in the first image, and the content included in the ROI is the content that the user is interested in. For example, when the acquisition scene of the first image is a security scene, the ROI generally refers to the area in the image that includes pedestrians, motor vehicles, and non-motor vehicles.

[0142] Optionally, the ROI information is obtained by identifying the first image through an artificial intelligence model. For example, dedicated AI hardware (such as a graphics processing unit (GPU) or a neural network processor (NPU)) can be deployed on the video encoding end, and the AI ​​model is run on these AI hardware to identify the ROI on the first image, thereby obtaining the ROI information of the first image. Specifically, the AI ​​model deployed on the video encoding end can be a semantic segmentation model, which can perform semantic segmentation on the image, thereby identifying and segmenting areas with specific semantics in the image. For example, in a security scenario, the semantic segmentation model can segment and mark areas in the image that include pedestrians, motor vehicles, or non-motor vehicles, thereby realizing ROI recognition of the image.

[0143] For example, see Figure 4 , Figure 4 A schematic diagram of performing ROI recognition on an image provided in an embodiment of the present application. Figure 4 As shown, the first image may be, for example, an image captured in a security scenario, and the image includes vehicles on the road and buildings beside the road, etc. After the first image is recognized by the AI ​​model, the ROI of the first image can be obtained, that is, the area where the vehicle in the first image is located.

[0144] Step 302: During encoding of the first image, the video encoder encapsulates the ROI information into the encoded data of the first image.

[0145] When the video encoding end encodes the first image in the video, the video encoding end can regard the ROI information of the first image as part of the content of the first image, and also encode the ROI information of the first image, thereby encapsulating the ROI information of the first image in the encoded data of the first image.

[0146] The ROI information is encapsulated in any of the following fields in the coded data: SEI field, IDR field, SPS field, PPS field or image coding slice field. As can be seen from the above introduction, the coded data in the code stream is composed of NALUs, and different NALUs may carry different types of data, and different types of data may be carried in the above-mentioned SEI field, IDR field, SPS field, PPS field or image coding slice field. Based on this, in this embodiment, the ROI information may be any of the above fields encapsulated in the coded data of the first image.

[0147] The SEI field usually does not carry image data information, but carries information that supplements and enhances the image data information or video stream. Since the ROI information itself is an expression of image semantic information, it enhances the understanding of the semantic information of the image itself, so it can be considered as additional enhanced information in image acquisition scenarios (such as security scenarios), so it is very appropriate to put the ROI information in the SEI field.

[0148] The IDR field is usually used to carry IDR frames. IDR frames refer to instant decoding refresh images. The purpose of IDR frames is to resynchronize decoding. When the decoder decodes an IDR frame, it immediately clears the reference frame queue, outputs or discards all decoded data, searches for the parameter set again, and starts a new sequence.

[0149] The SPS field refers to a sequence parameter set, which stores a set of global parameters of a video coding sequence, such as the coding profile, level, and image width and height. When carrying ROI information in the SPS field, the roi_flag flag in the SPS field can be reused to carry the ROI information; or, a roi_segmap flag can be extended in the SPS field, and the ROI information can be carried through the roi_segmap flag.

[0150] The PPS field refers to the image parameter set, and the parameters that the encoded data of each frame depends on are stored in the PPS. Similarly, in the PPS field, some of the original flags in the PPS field can be reused to carry ROI information, or a new flag can be extended to carry ROI, which will not be repeated here.

[0151] The picture coding slice field is usually used to carry the picture data of each picture coding slice. In which, a picture is usually divided into multiple picture coding slices, and each picture coding slice includes multiple macroblocks.

[0152] For example, see Figure 5 , Figure 5 A schematic diagram of encoding data of an image provided in an embodiment of the present application. Figure 5 As shown, during the encoding process, the image to be encoded is divided into two image coding slices, namely, image coding slice 1 and image coding slice 2. Therefore, in the encoded data of the image, it specifically includes a header type field, an SPS field for carrying SPS valid data, a PPS field for carrying PPS valid data, and an image coding slice field for carrying image coding slice valid data. In addition, the valid data of image coding slice 1 and the valid data of image coding slice 2 are respectively carried by two different image coding slice fields.

[0153] exist Figure 5 In the embodiment, the ROI information is carried in the image coding slice field, that is, after the image coding slice field carrying the valid data of the image coding slice 2, another image coding slice field is used to carry the ROI information.

[0154] It should be noted that Figure 5 In the figure, the ROI information is carried in the image coding slice field as an example. In actual applications, the ROI information can also be carried in the SPS field, the PPS field, the SEI field (not shown in the figure) or the IDR field (not shown in the figure), and this embodiment does not make any specific limitation on this.

[0155] Step 303: The video encoding end sends the encoded data of the first image to the video decoding end.

[0156] After encoding to obtain the encoded data of the first image, the video encoding end can send the encoded data of the first image to the video decoding end through the network. Exemplarily, in a security scenario, the video encoding end can be, for example, a collection module (such as a camera) deployed at the front end, and the video decoding end can be deployed at a storage module or a parsing module at the back end, wherein the storage module and the parsing module can be, for example, modules deployed on a server.

[0157] The encoded data sent by the video encoding segment is encapsulated with ROI information, and the ROI information is used to indicate the ROI in the first image.

[0158] It should be noted that, since the first image is an image in the video, the video encoding end actually continuously encodes each image in the video, thereby obtaining a code stream composed of the encoded data of each image. During the data transmission process, the video encoding end continuously transmits the code stream to the video decoding end, and the code stream includes the encoded data of the encoded image, that is, the encoded data of the first image mentioned above is part of the code stream.

[0159] Step 304: The video decoding end decodes the encoded data to obtain a first image and ROI information.

[0160] After receiving the encoded data, the video decoding end may decode the encoded data according to a video encoding standard corresponding to the encoded data, thereby obtaining the first image and the ROI information of the first image.

[0161] Since the first image is an image in the video, the ROI information of the first image obtained by the video decoding end by decoding the encoded data of the first image can be used by the video processing application to assist in performing the corresponding video processing task. Exemplarily, the ROI information of the first image can be used to perform any of the following tasks on the video where the first image is located: video compression, video tagging, live album, and video condensation.

[0162] Among them, video compression refers to the secondary compression of the video to obtain a video with a smaller data volume, which is convenient for video storage and transmission. After obtaining the ROI information of each image in the video, different compression parameters can be assigned to different areas in the same image based on the ROI information of each image, so as to achieve different compression strengths to compress different areas in the same image. For example, for the ROI in the image, a compression parameter with a smaller compression strength can be used to ensure that the quality of the content in the ROI of the image will not be greatly affected by compression; for the non-ROI in the image, a compression parameter with a larger compression strength can be used to ensure that the data volume of the compressed image is low.

[0163] Video marking refers to marking the video clips that need to be focused on in the video, so that the video clips that need to be focused on can be quickly found later. For example, in a security scenario, it is necessary to mark the video clips with more pedestrians, motor vehicles or non-motor vehicles on the road, so that the corresponding video positions can be quickly found when needed. Therefore, after obtaining the ROI information of each image in the video, the video clips with a larger ROI ratio or more ROI in the video can be marked, thereby achieving the purpose of video marking.

[0164] Live album refers to extracting video clips that need to be focused on or need to be processed later, so as to quickly obtain the video or image that the user is interested in from a longer video. In the process of executing live album, searching for video clips based on the ROI information of the image is similar to the video marking process. For details, please refer to the video marking process introduced above, which will not be repeated here.

[0165] Video condensation refers to condensing a long video into a shorter video, for example, condensing a 5-hour video into a 1-hour video. In the process of video condensation, the images in the video segments that do not need to be focused on can be identified, thereby achieving the purpose of video condensation. Specifically, based on the ROI of each image in the video, the video segments containing images without ROI or with a small ROI can be deleted to obtain a condensed video, and most of the duration in the condensed video corresponds to images with ROI or a large ROI.

[0166] In this embodiment, when encoding an image in a video, the ROI information of the acquired image is simultaneously encapsulated into the encoded data of the image, thereby realizing simultaneous transmission of the image and the ROI information of the image to the video decoding end, so that the video decoding end can further process the video based on the ROI information of the image, avoiding the video decoding end from re-identifying the ROI based on the decoded image, and ensuring the effective use of the ROI information of the image.

[0167] The above describes the process of encapsulating ROI information into the coded data of an image. The following describes in detail various methods of encapsulating ROI information into the coded data of an image with reference to specific examples.

[0168] Implementation method 1: encapsulate the ROI information in the form of a graph in the encoded data of the image.

[0169] Specifically, the ROI information encapsulated in the encoded data may be a ROI mask map, the ROI mask map includes a plurality of pixels, and the value of each pixel in the plurality of pixels is used to indicate whether each pixel belongs to the ROI.

[0170] That is, the ROI information of the first image is expressed in the form of a ROI mask map, that is, the value of each pixel in the ROI mask map indicates whether each pixel belongs to the ROI of the first image. In this way, by searching for pixels with specific values ​​in the ROI mask map, the position of the ROI of the first image can be determined. For example, in the ROI mask map, the value of a pixel located outside the ROI can be 0, and the value of a pixel located in the ROI can be a non-zero value.

[0171] In this solution, by representing the ROI information in the form of a ROI mask map, it is possible to clearly indicate whether each pixel in the image belongs to the ROI, and the ROI mask map can be used as a part of the image information. The ROI mask map is compressed and encoded in the same manner as the compressed encoded image, thereby minimizing the changes to the existing technology and ensuring that valid ROI information is carried in the encoded data of the image.

[0172] Optionally, the value of the target pixel in the ROI mask is also used to indicate the category of the ROI to which the target pixel belongs, and the target pixel is the pixel in the ROI mask that belongs to the ROI. That is to say, for the pixels located in the ROI, the values ​​of these pixels can indicate not only whether the pixels are located in the ROI, but also the category of the ROI in which these pixels are located.

[0173] For example, in a security scenario, for a pixel in the ROI mask map, a pixel value of 0 means that the pixel belongs to the background area, a pixel value of 1 means that the pixel belongs to the pedestrian area, a pixel value of 2 means that the pixel belongs to the motor vehicle area, and a pixel value of 3 means that the pixel belongs to the non-motor vehicle area. Therefore, in the ROI mask map, the area where the pixel with a value of 0 is located is a non-ROI, and the area where the pixel with a value other than 0 is located is a ROI. Moreover, although the pixel with a value of 1, 2, or 3 is in the ROI, the category of the ROI where the pixel with a value of 1 is located is a pedestrian area, the category of the ROI where the pixel with a value of 2 is located is a motor vehicle area, and the category of the ROI where the pixel with a value of 3 is located is a non-motor vehicle area.

[0174] In this scheme, by assigning different values ​​to different pixels in the ROI mask image, the category of the ROI to which the pixel belongs can be further distinguished based on the pixel value, thereby providing ROI information with richer semantic information and ensuring that the video decoding end can effectively use the ROI information to perform subsequent video processing tasks.

[0175] Optionally, the ROI information is also used to indicate the total number of ROI categories in the first image, and the value of the target pixel in the ROI mask image is related to the category of the ROI to which the target pixel belongs and the total number of ROI categories in the first image. The ROI information may directly indicate the total number of ROI categories in the first image; for example, in a traffic scene, the ROI categories include pedestrian areas, motor vehicle areas, and non-motor vehicle areas, and the ROI information may indicate that the total number of ROI categories in the first image is 3.

[0176] The ROI information may also directly indicate the total number of categories of the region in the first image. Since the first image necessarily includes non-ROIs, the total number of categories of the region in the first image is actually the total number of categories of the ROI in the first image plus 1. The total number of categories of the ROI in the first image can also be determined based on the total number of categories of the region in the first image indicated in the ROI information, that is, the ROI information indirectly indicates the total number of categories of the ROI in the first image.

[0177] Furthermore, in order to facilitate accurate determination of the category of the region to which each pixel belongs during the decoding process, the actual value of each pixel can be determined based on the category of the ROI to which the pixel belongs and the total number of categories of the ROI in the first image during the encoding process of the ROI mask map, that is, the value of the target pixel in the ROI mask map is related to the category of the ROI to which the target pixel belongs and the total number of categories of the ROI in the first image. For example, when the value range of the pixel in the ROI mask map is 0-255, and the ROI category to which the pixel belongs includes a pedestrian area, a motor vehicle area, and a non-motor vehicle area, it can be determined that there are 4 area categories corresponding to the pixel in the ROI mask map (i.e., non-ROI, pedestrian area, motor vehicle area, and non-motor vehicle area), so the value range (0-255) can be divided into four intervals, and a certain value in each interval (e.g., the center value of the interval) is used as the value corresponding to each category. Specifically, the value range of 0-255 can be divided into four intervals: 0-63, 64-127, 128-191, and 192-255; and, when the value of the pixel point is 32 (i.e., the center value of the interval 0-63), it represents that the region category to which the pixel point belongs is non-ROI; when the value of the pixel point is 96 (i.e., the center value of the interval 64-127), it represents that the region category to which the pixel point belongs is the pedestrian region; when the value of the pixel point is 160 (i.e., the center value of the interval 128-191), it represents that the region category to which the pixel point belongs is the motor vehicle region; when the value of the pixel point is 224 (i.e., the center value of the interval 192-255), it represents that the region category to which the pixel point belongs is the non-motor vehicle region.

[0178] In this scheme, by determining the actual value of the pixel in the ROI mask image based on the total number of ROI categories, the values ​​of the pixels belonging to different regions can be staggered as much as possible (that is, the difference between the values ​​is as large as possible), thereby ensuring that the quantization loss caused by the encoding process will not affect the determination of the region category to which each pixel belongs during the decoding process, thereby ensuring the feasibility of the scheme.

[0179] Optionally, the size of the ROI mask image is the same as the size of the first image. That is, the number of pixels included in the ROI mask image is the same as the number of pixels in the first image, and each pixel in the ROI mask image has a corresponding pixel in the first image. Based on the value of each pixel in the ROI mask image, it can be determined whether each pixel in the first image belongs to the ROI. By encapsulating the ROI mask image of the same size as the first image into the encoded data of the first image, the accuracy of the ROI information can be guaranteed, meeting the high accuracy requirements of the ROI information for subsequent video processing tasks.

[0180] Optionally, the ROI mask map may be obtained after downsampling, and the ROI mask map is smaller than the first image. For example, the ROI mask map encapsulated in the encoded data may be obtained after downsampling the original mask map, and the size of the original mask map is the same as the size of the first image, and each pixel in the original mask map is used to indicate the area to which the corresponding pixel in the first image belongs. For example, assuming that the size of the first image is 256*256, the size of the original mask map is also 256*256, and the ROI mask map can be 128*128 (that is, obtained after downsampling the original mask map by 2 times). Among them, the downsampling multiple corresponding to the ROI mask map can be determined according to actual requirements, and this embodiment does not limit this.

[0181] In this solution, by encapsulating the ROI mask image obtained after downsampling into the encoded data, the data volume of the ROI information can be greatly reduced while ensuring the accuracy of the ROI information as much as possible, reducing the data volume of the encoded data and the computing resources required for the encoding process.

[0182] For example, see Figure 6 , Figure 6 A schematic diagram of a processing flow of a ROI mask image provided in an embodiment of the present application. Figure 6 As shown in the figure, at the video encoding end, after obtaining the ROI mask map input by the AI ​​model, the ROI mask map can be converted into an image in YUV400 format, and the video encoding process is performed on the image in YUV400 format to obtain a bitstream and transmit the bitstream to the video decoding end. After obtaining the bitstream transmitted by the video encoding end, the video decoding end decodes the bitstream to obtain an image in YUV400 format; then the image in YUV400 format is converted into an ROI mask map, thereby outputting the ROI mask map, so as to perform corresponding video processing tasks based on the ROI mask map.

[0183] See also Figure 7 , Figure 7 A schematic diagram of a ROI mask image provided by an embodiment of the present application following the transmission of an I frame and a P frame. Figure 7 As shown, during the video encoding process, I frames and P frames are transmitted alternately. Usually, after transmitting one I frame, 7 P frames are transmitted, and then the next I frame is transmitted, and this cycle is repeated. In this embodiment, when transmitting each frame, the corresponding ROI mask image can be additionally encapsulated on the original encoded data of the frame, so as to realize encapsulating the ROI information corresponding to each encoded image in the video.

[0184] Generally speaking, the ROI mask map identified by the AI ​​model is a map of size w*h, and each pixel in the ROI mask map takes an integer value such as 0, 1, 2, 3, etc. For example, in a security scenario, a pixel value of 0 indicates that the pixel belongs to the background area, a pixel value of 1 indicates that the pixel belongs to the pedestrian area, a pixel value of 2 indicates that the pixel belongs to the motor vehicle area, and a pixel value of 3 indicates that the pixel belongs to the non-motor vehicle area. The background area belongs to the non-ROI, and the pedestrian area, motor vehicle area, and non-motor vehicle area all belong to the ROI, so the value of a pixel can be represented by 2 bits of information.

[0185] Then, for each pixel value x_i (x_i is represented by k bits of information) of the ROI mask image, shift it left by 8-k bits and fill it into 8 bits of information. For example, when 0, 1, 2, 3 are used to represent the value of the pixel point, the pixel value is shifted left by 6 bits, 0 represents the background, 64 represents a person, 128 represents a motor vehicle, and 192 represents a non-motor vehicle. Specifically, in the binary algorithm, 8 bits represent that the value range of a single pixel value is between [0,255], that is, 2 8 =256. 8-k is to convert the original 0-1-2-3 into -0-64-128-192, which has data distribution and is convenient for image encoding. For example, please refer to Figure 8 , Figure 8 A schematic diagram of an embodiment of the present application providing a method of shifting the pixel values ​​in the ROI mask image left by 8-k bits. Figure 8 As shown, the initial value of the pixel in the ROI mask image is represented by 2 bits of information. For a pixel whose initial value is 01, its value is shifted left by 6 bits to 01000000.

[0186] See also Fig. 9 , Fig. 9 A schematic diagram of a packaged ROI mask image provided in an embodiment of the present application. Fig. 9 As shown, in the encoded data of the image, since the ROI mask map of the image is essentially an image, the ROI mask map of the image can be specifically encapsulated in the IDR field to ensure that the changes to the existing technology are reduced as much as possible.

[0187] In addition, in the Video Parameter Set (VPS) field of the encoded data, the following multiple parts of content may be added as part of the ROI information to further describe the situation of the ROI mask map.

[0188] The first part: the size of the ROI mask.

[0189] The size of the ROI mask map can be represented by two 16-bit unsigned integers, segmap_w and segmap_h. Segmap_w is used to indicate the width of the ROI mask map, and segmap_h is used to indicate the height of the ROI mask map. Based on segmap_w and segmap_h, the size of the ROI mask map can be determined, for example, the size of the ROI mask map is 256*128.

[0190] The second part: the number of skipped frames for ROI segmentation (seg_skip_num).

[0191] The number of frame skips for ROI segmentation can be represented by an 8-bit unsigned integer, which specifically refers to the number of image frames in the video at which ROI segmentation is performed, that is, ROI information is encoded once every N image frames. On some devices with limited encoding capabilities or ROI segmentation capabilities, the ROI information corresponding to each image frame may not be encoded for each image frame, so the ROI information may be encoded once every multiple image frames. Specifically, the number of frame skips for ROI segmentation can be determined based on actual conditions, and this embodiment does not make any specific restrictions on this. Figure 7 As shown, Figure 7 The number of frame skipping of the ROI segmentation corresponding to the video encoded in is 0, that is, the corresponding ROI information is encoded for each image frame.

[0192] The third part: the total number of regional categories in the ROI mask map (category_num).

[0193] The total number of region categories in the ROI mask map may be represented by an 8-bit unsigned integer, specifically referring to the total number of categories of all regions included in the ROI mask map. For example, when the regions included in the ROI mask map include non-ROI, pedestrian regions, motor vehicle regions, and non-motor vehicle regions, the total number of region categories in the ROI mask map is 4.

[0194] When encoding the ROI mask map, the actual value of each pixel can be determined according to the total number of region categories of the ROI mask map and the region category to which the pixel in the ROI mask map belongs. For example, when category_num is 4 and the value range of the pixel is 0-255, the value of the pixel in the region belonging to category 0 can be 256 / 4*0+32=32; the value of the pixel in the region belonging to category 1 can be 256 / 4*1+32=96; the value of the pixel in the region belonging to category 2 can be 256 / 4*2+32=160; the value of the pixel in the region belonging to category 3 can be 256 / 4*3+32=224.

[0195] It should be noted that, in the process of decoding the encoded data, the ROI mask map in the encoded data can be decoded first to obtain the value of each pixel in the ROI mask map. Due to the quantization loss of the encoding process, the value of the pixel obtained by decoding may be different from the value corresponding to the category to which the pixel actually belongs. Therefore, multiple value ranges are defined (i.e., a certain redundancy value is set), and based on the target value range of the value of each pixel in the multiple value ranges, the category of the area to which each pixel belongs is determined. Among them, the multiple value ranges are determined based on the total number of categories of the ROI in the first image.

[0196] For example, when category_num is 4 and the value range of the pixel is 0-255, the total value range (0-255) can be divided into four small value ranges, so as to determine the region category to which the pixel belongs based on the value range of the actual value of the pixel. In this way, if the value of the pixel obtained by decoding falls into the value range [n*256 / category_num, (n+1)*256 / category_num), the category of the region to which the pixel belongs is determined to be the nth category. For example, category_num=4, the corresponding value decoded for a pixel is 187, which is between [2*256 / 4, 3*256 / 4), so the region category to which the pixel belongs is the second category.

[0197] Implementation method 2: encapsulate the ROI information in the form of a point set in the encoded data of the image.

[0198] Specifically, the ROI information encapsulated in the encoded data includes one or more point sets, each point set includes multiple points, and the closed area surrounded by the multiple points in each point set belongs to the ROI in the first image. In other words, a point set is used to represent a closed area, and the closed area represented by each point set belongs to the ROI in the image, that is, the area of ​​the image belonging to the ROI is represented by multiple points located on the edge of the area. Generally speaking, the number of point sets in the ROI information corresponds to the number of ROIs included in the image. The more separated ROIs in the image, the more point sets there are.

[0199] The one or more point sets may be calculated after the AI ​​model outputs the ROI of the recognized image. Fig.10 , Fig.10 A schematic diagram of representing the ROI of an image by a set of points provided in an embodiment of the present application. Fig.10As shown, after determining the ROI of the image, for each area belonging to the ROI (such as the area where non-motor vehicles are located, the area where motor vehicles are located, and the area where pedestrians are located), the edges of these areas can be determined by algorithms, and the pixels located on the edges of these areas can be determined. In this way, each area belonging to the ROI in the image can be represented by the pixels on the edge of the area, that is, the closed area surrounded by the pixels on the edge of the area is the area currently required to be represented.

[0200] In this scheme, by using a set of points that can enclose a closed area to represent the ROI in the image, it is possible to represent a larger ROI in the image with a small number of points, avoiding the use of all pixels in the ROI to represent the ROI, and effectively representing the ROI in the image in the form of low data volume, thereby ensuring the feasibility of the scheme.

[0201] Optionally, the ROI information encapsulated in the encoded data is also used to indicate the category of the ROI to which the closed area surrounded by each point set belongs. Specifically, the ROI information can indicate the category of the ROI to which the closed area surrounded by each point set belongs through a specific flag bit. For example, the flag bit can take values ​​such as 1, 2, and 3 to indicate that the category of the ROI to which the closed area surrounded by each point set belongs is a pedestrian area, a motor vehicle area, or a non-motor vehicle area.

[0202] For example, see Fig.11 , Fig.11 A schematic diagram of a processing flow of a point set in ROI information provided in an embodiment of the present application. Fig.11 As shown, at the video encoding end, after obtaining the ROI mask map input by the AI ​​model, the ROI mask map can be converted into one or more point sets, so as to realize the ROI of the image based on one or more point sets. After obtaining one or more point sets based on the ROI mask map, in order to realize the transmission of point set data, corresponding point structure data can be generated based on one or more point sets, so as to realize the carrying of ROI information in the encoded data. After obtaining the point structure data, the point structure data can be quantized to obtain quantized point structure data; then, the quantized point structure data is entropy encoded to obtain a bitstream and transmit the bitstream to the video decoding end. After obtaining the bitstream transmitted by the video encoding end, the video decoding end entropy decodes the bitstream to obtain quantized point structure data; then, the quantized point structure data is dequantized to obtain point structure data, and finally the point structure data is converted into a ROI mask map, thereby outputting the ROI mask map, so as to perform corresponding video processing tasks based on the ROI mask map.

[0203] See also Fig.12 , Fig.12A schematic diagram of a packaging point structure data provided by an embodiment of the present application. Fig.12 As shown, in the encoded data of the image, the point structure data used to represent the ROI information can be specifically encapsulated in the SEI field, that is, the point structure data plays a supplementary and enhancing role to avoid affecting other fields carrying image information, thereby ensuring that changes to the existing technology are minimized as much as possible.

[0204] For ease of understanding, the specific representation of point structure data will be introduced below. Specifically, the point structure data may include multiple parts of data as described below.

[0205] 1. Region of interest enable flag (roi_enable)

[0206] The ROI enable flag is a 1-bit unsigned integer used to indicate whether the ROI information of the current image is enabled. If the ROI enable flag is 0, it means that the current image has no ROI information, and if the ROI enable flag is 1, it means that the current image has ROI information.

[0207] 2. Number of regions of interest (roi_num)

[0208] The number of regions of interest is an 8-bit unsigned integer used to represent the number of regions of interest in the current image.

[0209] 3. Region of interest information structure (ROI_Info)

[0210] The region of interest information structure is a multi-bit unsigned integer, which is used to represent a structure containing a region of interest information of the current image, including fields such as category, prob, point_num, points, etc. Generally, all region of interest information of the current image is represented by an ROI_Info array.

[0211] 4. Region of interest category information (ROIInfo[i]->category)

[0212] The category information of the region of interest is an 8-bit unsigned integer, which is used to represent the category information of the i-th region of interest of the current image. Generally, a value of 0 is used to represent the background category.

[0213] 5. Region of interest probability value (ROIInfo[i]->prob)

[0214] The probability value of the region of interest is an 8-bit unsigned integer, which is used to represent the category probability value of the i-th region of interest in the current image. A probability value of 0 means that it is not effective and there is no probability value information; when the probability value of the region of interest takes other values, the probability value represented can be agreed upon by the user.

[0215] 6. Number of polygon points in the region of interest (ROIInfo[i]->point_num)

[0216] The region of interest probability value is an 8-bit unsigned integer, which is used to represent the number of polygon points in the i-th region of interest of the current image.

[0217] 7. Region of interest polygon point information (ROIInfo[i]->points)

[0218] The polygon point information of the region of interest is a multi-bit unsigned integer, which is used to represent a point information of the polygon of the ith region of interest. Generally, a region of interest information is represented by a points array.

[0219] 8. Difference of the horizontal coordinates of the points in the region of interest (ROIInfo[i]->points[j]->x)

[0220] The horizontal coordinate difference of the ROI point is an 8-bit signed integer, which is used to represent the horizontal coordinate difference between the j-th point and the j-1-th point in the i-th ROI of the current image. For the 0th point, its horizontal coordinate value is points[0]->x+128. In order to reduce the bit rate increase caused by ROI information, the coordinate value is limited to [-128,127]. The mapping relationship between this value and the real coordinate can be agreed upon by the user.

[0221] 9. The difference in ordinate of the point in the region of interest (ROIInfo[i]->points[j]->y)

[0222] The ordinate difference of the point in the region of interest is an 8-bit signed integer, which is used to represent the ordinate difference between the j-th point and the j-1-th point in the i-th region of interest of the current image. For the 0th point, its ordinate value is points[0]->y+128.

[0223] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of the method. It is understandable that, in order to realize the above functions, the electronic device includes a hardware structure and / or software module corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the modules and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0224] The embodiment of the present application can divide the functional modules of the electronic device according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0225] The following is a detailed description of the device used to execute the above method in the embodiment of the present application.

[0226] See also Fig.13 , Fig.13 FIG. 1 is a schematic diagram of the structure of a video encoding device provided in an embodiment of the present application. Fig.13 As shown, the video encoding device includes: an acquisition module 1301, used to obtain ROI information of a region of interest, where the ROI information is used to indicate an ROI in a first image, where the first image is an image in a video to be encoded; an encoding module 1302, used to encapsulate the ROI information in the encoding data of the first image during encoding of the first image; and a sending module 1303, used to send the encoding data of the first image.

[0227] In a possible implementation, the ROI information is encapsulated in any one of the following fields in the coded data: a supplemental enhancement information SEI field, an instantaneous decoding refresh IDR field, a sequence parameter set SPS field, a picture parameter set PPS field, or a picture coded slice field.

[0228] In a possible implementation, the ROI information encapsulated in the encoded data includes a ROI mask map, the ROI mask map includes a plurality of pixels, and the value of each pixel in the plurality of pixels is used to indicate whether each pixel belongs to the ROI.

[0229] In a possible implementation, the value of the target pixel in the ROI mask image is also used to indicate the category of the ROI to which the target pixel belongs. The target pixel is a pixel in the ROI mask image that belongs to the ROI.

[0230] In a possible implementation, the ROI information is also used to indicate the total number of ROI categories in the first image, and the value of the target pixel in the ROI mask image is related to the category of the ROI to which the target pixel belongs and the total number of ROI categories in the first image.

[0231] In a possible implementation manner, the ROI mask image is obtained after downsampling, and the ROI mask image is smaller than the first image.

[0232] In a possible implementation, the ROI information encapsulated in the encoded data includes one or more point sets, each point set includes multiple points, and a closed area surrounded by the multiple points in each point set belongs to the ROI in the first image.

[0233] In a possible implementation, the ROI information encapsulated in the encoded data is also used to indicate the category of the ROI to which the closed area surrounded by each point set belongs.

[0234] In a possible implementation, the ROI information is obtained by identifying the first image through an artificial intelligence model.

[0235] In a possible implementation manner, the acquisition scene of the first image is a security scene.

[0236] See also Fig.14 , Fig.14 FIG. 1 is a schematic diagram of the structure of a video decoding device provided in an embodiment of the present application. Fig.14 As shown, the video decoding device includes: a receiving module 1401, used to receive the encoded data of the first image, the encoded data is encapsulated with ROI information, the ROI information is used to indicate the ROI in the first image, and the first image is an image in the video; a decoding module 1402, used to decode the encoded data to obtain the first image and the ROI information.

[0237] In a possible implementation, the ROI information is used to perform any one of the following tasks on the video: video compression, video tagging, live album, and video condensation.

[0238] In a possible implementation manner, the ROI information is encapsulated in any one of the following fields in the coded data: a SEI field, an IDR field, an SPS field, a PPS field, or an image coding slice field.

[0239] In a possible implementation, the ROI information encapsulated in the encoded data includes a ROI mask map, the ROI mask map includes a plurality of pixels, and the value of each pixel in the plurality of pixels is used to indicate whether each pixel belongs to the ROI.

[0240] In a possible implementation, the value of the target pixel in the ROI mask image is also used to indicate the category of the ROI to which the target pixel belongs. The target pixel is a pixel in the ROI mask image that belongs to the ROI.

[0241] In a possible implementation manner, the ROI information is further used to indicate the total number of categories of ROIs in the first image;

[0242] The decoding module 1402 is also used to: decode the ROI mask image in the encoded data to obtain the value of each pixel in the ROI mask image; determine the category of the area to which each pixel belongs based on the target value range of the value of each pixel in multiple value ranges; wherein the multiple value ranges are determined based on the total number of categories of ROI in the first image.

[0243] In a possible implementation manner, the ROI mask image is obtained after downsampling, and the ROI mask image is smaller than the first image.

[0244] In a possible implementation, the ROI information encapsulated in the encoded data includes one or more point sets, each point set includes multiple points, and a closed area surrounded by the multiple points in each point set belongs to the ROI in the first image.

[0245] In a possible implementation, the ROI information encapsulated in the encoded data is also used to indicate the category of the ROI to which the closed area surrounded by each point set belongs.

[0246] In a possible implementation, the ROI information is obtained by identifying the first image through an artificial intelligence model.

[0247] In a possible implementation manner, the acquisition scene of the first image is a security scene.

[0248] See also Fig.15 , Fig.15 FIG. 1 is a schematic diagram of the structure of a chip according to an embodiment of the present application. Fig.15 As shown, the chip may include hardware modules such as a hardware decoder, a hardware image processor, a neural network processor (neural-network processing unit, NPU), a central processing unit (central processing unit, CPU) and a hardware encoder, which can support the encoding and decoding of the video. Wherein, when the chip is deployed at the video encoding end, the hardware encoder in the chip can be used to execute the video encoding method provided in the embodiment of the present application. When the chip is deployed at the video decoding end, the hardware decoder in the chip can be used to execute the video decoding method provided in the embodiment of the present application.

[0249] It should be noted that the information interaction, execution process, etc. between the modules / or components in the above device are the same as those in this application. Figure 3-Figure 12 The corresponding method embodiments are based on the same concept. For specific contents, please refer to the description in the method embodiments shown above in this application, which will not be repeated here.

[0250] It should be noted that for the specific implementation of the device and the beneficial effects brought about, please refer to Figure 3-Figure 12 The descriptions in the corresponding method embodiments will not be repeated here one by one.

[0251] An embodiment of the present application also provides a computer-readable storage medium, including instructions, which, when executed on a computer, enables the computer to control an electronic device to execute any one of the implementation methods shown in the aforementioned method embodiments.

[0252] An embodiment of the present application also provides a computer program product, which includes a computer program code. When the computer program code runs on a computer, the computer executes any one of the implementation methods shown in the aforementioned method embodiments.

[0253] An embodiment of the present application also provides a chip system, including a memory and a processor, the memory is used to store computer programs, and the processor is used to call and run computer programs from the memory, so that the chip executes any one of the implementation methods shown in the aforementioned method embodiments.

[0254] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0255] Through the description of the above implementation methods, the technicians in the relevant field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structures used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation method in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, U disk, mobile hard disk, ROM, RAM, disk or optical disk, etc., including a number of instructions to enable a computer device to execute the methods described in each embodiment of the present application.

[0256] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0257] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a communication device, a computing device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, communication device, computing device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a communication device, a data center, or other data storage device that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

[0258] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0259] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

Claims

1. A video encoding method, characterized in that: include: Acquire region of interest ROI information, where the ROI information is used to indicate a ROI in a first image, where the first image is an image in a video to be encoded; In a process of encoding the first image, encapsulating the ROI information in the encoded data of the first image; The encoded data of the first image is transmitted.

2. The method according to claim 1, characterized in that The ROI information is encapsulated in any one of the following fields in the coded data: a supplemental enhancement information SEI field, an instant decoding refresh IDR field, a sequence parameter set SPS field, a picture parameter set PPS field or a picture coded slice field.

3. The method according to claim 1 or 2, characterized in that: The ROI information encapsulated in the encoded data includes a ROI mask map, the ROI mask map includes a plurality of pixels, and a value of each pixel in the plurality of pixels is used to indicate whether the each pixel belongs to the ROI.

4. The method according to claim 3, characterized in that The value of the target pixel in the ROI mask image is also used to indicate the category of the ROI to which the target pixel belongs. The target pixel is a pixel belonging to the ROI in the ROI mask image.

5. The method according to claim 4, characterized in that The ROI information is also used to indicate the total number of ROI categories in the first image, and the value of the target pixel in the ROI mask image is related to the category of the ROI to which the target pixel belongs and the total number of ROI categories in the first image.

6. The method according to any one of claims 3 to 5, characterized in that: The ROI mask image is obtained after downsampling, and the ROI mask image is smaller than the first image.

7. The method according to claim 1 or 2, characterized in that: The ROI information encapsulated in the encoded data includes one or more point sets, each point set includes multiple points, and the closed area surrounded by the multiple points in each point set belongs to the ROI in the first image.

8. The method according to claim 7, characterized in that The ROI information encapsulated in the encoded data is also used to indicate the category of the ROI to which the closed area surrounded by each point set belongs.

9. The method according to any one of claims 1 to 8, characterized in that: The ROI information is obtained by identifying the first image through an artificial intelligence model.

10. The method according to any one of claims 1 to 9, characterized in that: The acquisition scene of the first image is a security scene.

11. A video decoding method, characterized in that: include: Receive encoded data of a first image, wherein the encoded data encapsulates ROI information, and the ROI information is used to indicate a ROI in the first image, where the first image is an image in a video; The encoded data is decoded to obtain the first image and the ROI information.

12. The method according to claim 11, characterized in that The ROI information is used to perform any one of the following tasks on the video: video compression, video tagging, live album, and video condensation.

13. The method according to claim 11 or 12, characterized in that: The ROI information is encapsulated in any one of the following fields in the coded data: a SEI field, an IDR field, a SPS field, a PPS field, or a picture coding slice field.

14. The method according to any one of claims 11 to 13, characterized in that: The ROI information encapsulated in the encoded data includes a ROI mask map, the ROI mask map includes a plurality of pixels, and a value of each pixel in the plurality of pixels is used to indicate whether the each pixel belongs to the ROI.

15. The method according to claim 14, characterized in that The value of the target pixel in the ROI mask image is also used to indicate the category of the ROI to which the target pixel belongs. The target pixel is a pixel belonging to the ROI in the ROI mask image.

16. The method according to claim 14 or 15, characterized in that The ROI information is also used to indicate the total number of categories of ROIs in the first image; The decoding of the encoded data comprises: Decoding the ROI mask image in the encoded data to obtain a value of each pixel in the ROI mask image; Determine the category of the region to which each pixel belongs based on the target value range in which the value of each pixel falls among multiple value ranges; The multiple value ranges are determined based on the total number of categories of ROIs in the first image.

17. The method according to any one of claims 10 to 12, characterized in that: The ROI information encapsulated in the encoded data includes one or more point sets, each point set includes multiple points, and the closed area surrounded by the multiple points in each point set belongs to the ROI in the first image.

18. The method according to claim 17, characterized in that The ROI information encapsulated in the encoded data is also used to indicate the category of the ROI to which the closed area surrounded by each point set belongs.

19. A video encoding device, characterized in that: include: An acquisition module, used for acquiring region of interest ROI information, where the ROI information is used for indicating the ROI in a first image, where the first image is an image in a video to be encoded; an encoding module, configured to encapsulate the ROI information into encoding data of the first image during encoding of the first image; A sending module is used to send the encoded data of the first image.

20. A video decoding device, characterized in that: include: A receiving module, used for receiving encoded data of a first image, wherein the encoded data is encapsulated with ROI information, and the ROI information is used for indicating an ROI in the first image, and the first image is an image in a video; A decoding module is used to decode the encoded data to obtain the first image and the ROI information.

21. A computer storage medium, characterized in that The computer storage medium stores instructions, which, when executed by a computer, cause the computer to implement the method of any one of claims 1 to 18.

22. A computer program product, characterized in that The computer program product stores instructions, which, when executed by a computer, cause the computer to implement the method according to any one of claims 1 to 18.

Images