Coding method and device, decoding method and device, electronic equipment and storage medium

Abstract

The invention provides a coding method and device, a decoding method and device, electronic equipment and a storage medium. The coding method comprises the following steps: acquiring a video image; detecting the ROI of the video image based on an ROI detection model to obtain ROI information of the video image; based on the ROI information of the video image, extracting the ROI of the video image to obtain an ROI image; encoding the ROI image to obtain an ROI image encoding result; and packaging the ROI information of the video image and the ROI image coding result into an SEI auxiliary stream packet. According to the coding method, the coding effect of the ROI region can be reserved to the greatest extent, but not only the ROI category is saved, so that the coding effect of the ROI region of the video image is improved.

Description

TECHNICAL FIELD

[0001] The present disclosure belongs to the technical field of video coding, and particularly relates to an encoding method and device, a decoding method and device, an electronic device, and a storage medium. BACKGROUND

[0002] In the fields of ultra-clear video, cloud game, VR video, mobile live broadcast, intelligent monitoring, and unmanned aerial vehicle flight control, there is a demand for intelligent encoding for specific scenarios and specific targets. For example, hybrid encoding technology is used in some technologies / products to save network bandwidth; GameStream uses AI to dynamically adjust texture encoding; the ROI (Region of Interest) encoding of a live broadcast platform optimizes the stream quality of a mobile terminal; a monitoring encoder dynamically adjusts the key frame interval through AI detection. SUMMARY

[0003] The present disclosure provides an encoding method and device, a decoding method and device, an electronic device, and a storage medium, and is used to solve the problem of poor encoding effect of a video image ROI region in the prior art.

[0004] In a first aspect, the present disclosure provides an encoding method. The encoding method comprises: acquiring a video image; detecting a ROI of the video image based on a ROI detection model to acquire ROI information of the video image, wherein the ROI information of the video image comprises ROI position information and a ROI category; extracting the ROI of the video image based on the ROI information of the video image to acquire a ROI image; encoding the ROI image to acquire a ROI image encoding result; and packing the ROI information of the video image and the ROI image encoding result into an SEI auxiliary stream packet.

[0005] In an embodiment of the present disclosure, detecting the video image based on a ROI detection model to acquire ROI information of the video image comprises: loading the ROI detection model, and detecting the video image based on the ROI detection model to acquire the ROI information of the video image; when a ROI associated with ROI information of a plurality of previous frames of the video image disappears in the ROI information of the video image, performing motion estimation processing on the video image based on the ROI information of the plurality of previous frames of the video image to acquire estimated ROI position information of the disappeared ROI in the video image; and updating the estimated ROI position information and a category of the disappeared ROI to the ROI information of the video image.

[0006] In one embodiment of this disclosure, the ROI location information includes ROI size and ROI coordinates, and detecting the ROI of the video image based on the Region of Interest (ROI) detection model to obtain the ROI information of the video image further includes: filtering the ROI information of the video image based on the ROI size in the ROI information to obtain filtered ROI information; aligning the ROI coordinates in the filtered ROI information to obtain aligned ROI information; and obtaining... The structural similarity index and absolute difference sum of ROIs of the same category as those in the previous frame of the video image are taken from the aligned ROI information. The ROIs in the previous frame of the video image have their corresponding sequential positions. Based on the structural similarity index and the absolute difference sum, the ROIs in the aligned ROI information are sorted to obtain the sequential positions of the ROIs in the aligned ROI information. Based on the sequential positions of the ROIs in the aligned ROI information, the ROIs in the aligned ROI information are spliced ​​to obtain spliced ​​ROI information.

[0007] In one embodiment of this disclosure, sorting ROIs in the aligned ROI information based on the structural similarity index and the sum of absolute differences includes: determining the correlation between the ROIs in the aligned ROI information and the ROIs in the previous frame of the video image based on the structural similarity index and the sum of absolute differences; and if there are two highly correlated ROIs in the video image and the previous frame of the video image, then the two highly correlated ROIs are sorted in the same order; otherwise, based on the ROI position information in the aligned ROI information, the ROIs with low correlation are sorted.

[0008] In one embodiment of this disclosure, sorting ROIs with low relevance in the aligned ROI information based on ROI position information includes: sorting ROIs with low relevance in the aligned ROI information based on ROI size to obtain a first sorted ROI; and sorting the first sorted ROIs based on ROI coordinates to obtain a second sorted ROI, wherein the order position of the second sorted ROI is the order position of the ROI in the aligned ROI information.

[0009] In one embodiment of this disclosure, the encoding method further includes: downsampling the video image to obtain a low-resolution video image; encoding the low-resolution video image to obtain a bitstream of the low-resolution video image; and packaging the bitstream and the SEI auxiliary stream packet into a predetermined transmission format.

[0010] In one embodiment of this disclosure, extracting the ROI of the video image based on the ROI information of the video image to obtain an ROI image includes: defining an ROI level, wherein the ROI level is used to assign pixel ownership to overlapping regions in the ROI region according to the ROI level; and performing ROI extraction processing on the video image based on the ROI level and the ROI information of the video image to obtain the ROI image.

[0011] Secondly, this disclosure provides a decoding method. The decoding method includes: acquiring an SEI auxiliary stream packet and a bitstream of a video image; decoding the SEI auxiliary stream packet to obtain ROI information and an ROI image of the video image; decoding the bitstream to obtain the video image; and generating an enhanced video image based on the ROI information, the ROI image, and the video image.

[0012] In one embodiment of this disclosure, the decoding method further includes: reconstructing the video image, which is a low-resolution video image, using an AI model to obtain a high-resolution video image.

[0013] In one embodiment of this disclosure, generating an enhanced video image based on the ROI information, the ROI image, and the video image includes: mapping the ROI to a matching position in the high-resolution video image based on the ROI information; and mapping the ROI image onto the high-resolution video image with a gradient edge based on the matching position to generate the enhanced video image.

[0014] In one embodiment of this disclosure, the decoding method further includes: performing bilateral filtering and mean filtering on the high-frequency and low-frequency portions of the ROI boundary in the ROI image or the enhanced video image, respectively, to obtain a filtered image.

[0015] Thirdly, this disclosure provides an encoding apparatus. The encoding apparatus includes: a video image acquisition module configured to acquire a video image; a video image detection module configured to detect the Region of Interest (ROI) of the video image based on a Region of Interest (ROI) detection model to obtain ROI information of the video image, the ROI information including ROI location information and ROI category; an ROI image extraction module configured to extract the ROI of the video image based on the ROI information to obtain an ROI image; an ROI image encoding module configured to encode the ROI image to obtain an ROI image encoding result; an ROI information packaging module configured to package the ROI information of the video image and the ROI image encoding result into a Supplemental Enhancement Information (SEI) auxiliary stream packet; and a bitstream packaging module configured to package the bitstream based on the video image and the SEI auxiliary stream packet into a predetermined transmission format.

[0016] Fourthly, this disclosure provides a decoding apparatus. The decoding apparatus includes: a bitstream acquisition module configured to acquire a bitstream of an SEI auxiliary stream packet and a video image; an auxiliary stream packet parsing module configured to decode the SEI auxiliary stream packet to acquire ROI information and an ROI image of the video image; a bitstream parsing module configured to decode the bitstream to acquire the video image; and an image generation module configured to generate an enhanced video image based on the ROI information, the ROI image, and the video image.

[0017] Fifthly, embodiments of this disclosure also provide a computer-readable storage medium having a computer program stored thereon. When executed by a processor, the computer program implements the encoding method described in any of the first aspects and / or the decoding method described in any of the second aspects.

[0018] Fourthly, embodiments of this disclosure also provide an electronic device. The electronic device includes: a memory configured to store information associated with an image; and a processor electrically coupled to the memory and configured to perform the encoding method described in any of the first aspects and / or the decoding method described in any of the second aspects.

[0019] According to the encoding method and apparatus, decoding method and apparatus, electronic device and storage medium disclosed herein, in the encoding method, by detecting and extracting the ROI of the video image, further encoding the ROI image, and packaging the ROI information and the ROI image encoding result into the SEI auxiliary stream packet, the encoding effect of the ROI region can be preserved to the greatest extent, rather than just saving the ROI category, thereby improving the encoding effect of the ROI region of the video image. Attached Figure Description

[0020] Figure 1The diagram shown is a schematic representation of a smart coding architecture based on auxiliary flow enhancement according to an embodiment of the present disclosure.

[0021] Figure 2 The flowchart shown is an example of an encoding method according to an embodiment of the present disclosure.

[0022] Figure 3 The flowchart shown is a process for detecting the Region of Interest (ROI) of a video image according to an embodiment of the present disclosure using a Region of Interest (ROI) detection model to obtain ROI information of the video image.

[0023] Figure 4 The flowchart shown is a process for detecting the Region of Interest (ROI) of a video image according to an embodiment of the present disclosure using a Region of Interest (ROI) detection model to obtain ROI information of the video image.

[0024] Figure 5 The flowchart shown is an example of an encoding method according to an embodiment of the present disclosure.

[0025] Figure 6 The flowchart shown is a process of extracting the ROI of a video image based on the ROI information of the video image according to an embodiment of the present disclosure to obtain an ROI image.

[0026] Figure 7 The flowchart shown is a decoding method according to an embodiment of the present disclosure.

[0027] Figure 8 The flowchart shown is a process for generating an enhanced video image based on the ROI information, the ROI image, and the video image according to an embodiment of the present disclosure.

[0028] Figure 9 The diagram shown is a schematic representation of an encoding device according to an embodiment of the present disclosure.

[0029] Figure 10 The diagram shown is a schematic representation of a decoding apparatus according to an embodiment of the present disclosure. Detailed Implementation

[0030] The following specific examples illustrate the implementation of this disclosure. Those skilled in the art can easily understand other advantages and effects of this disclosure from the content disclosed in this specification. This disclosure can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this disclosure. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0031] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this disclosure. Therefore, the illustrations only show the components related to this disclosure and are not drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0032] The technical solutions of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0033] Current intelligent coding schemes use traditional pixel difference calculations, which are not very accurate. While they can solve some problems, they can also produce misjudgments in certain extreme scenarios, whereas AI has better generalization capabilities. Current intelligent coding schemes have basically reached the upper limit of traditional algorithms, and without the support of AI, it will be difficult to achieve significant breakthroughs in profitability.

[0034] Intelligent coding also faces several challenges: Problem 1) Computational overhead: The inference time of neural networks may offset the network latency saved by the bitrate; Problem 2) Hardware compatibility: There are significant differences between existing chip hardware and software, and there is no universally compatible AI model or algorithm; Problem 3) Standard compatibility: It is difficult for intelligent coding optimization to be compatible with the H264 / H265 standard; Problem 4) Video quality drift: The non-linear characteristics of neural networks may lead to bitrate fluctuations or subjective quality instability.

[0035] In the encoding method described in this disclosure, by identifying and extracting the ROI of the video image, further encoding the ROI image, and packaging the ROI information and the ROI image encoding result into the SEI auxiliary stream packet, the encoding effect of the ROI region can be preserved to the greatest extent, rather than just saving the ROI category, thereby improving the encoding effect of the ROI region of the video image.

[0036] In some embodiments, by supporting end-to-end intelligent encoding, the computational load of the intelligent encoder and decoder is fully balanced, thereby avoiding problem 1). Optional AI model dependency allows the heavy-load AI model to be used as an optional add-on, thereby avoiding problems 2) and 4). Full utilization of SEI features maintains compatibility with popular codec ecosystems, thereby avoiding problem 3). The encoding method of this disclosure can reduce the bitrate by approximately 40% based on H.264 / H.265 encoding while maintaining the same image quality, making it suitable for high-resolution but low-bitrate application scenarios.

[0037] The following will elaborate on the principles and implementation methods of the encoding method and apparatus, decoding method and apparatus, electronic equipment and storage medium of this disclosure, so that those skilled in the art can understand the technical solutions of this disclosure without creative effort.

[0038] Figure 1 This is a schematic diagram illustrating the structure of an intelligent coding architecture based on auxiliary stream enhancement according to an embodiment of the present disclosure. The intelligent coding architecture includes the following core components: an intelligent stream encoder, an intelligent stream decoder, ROI region identification and encoding, ROI region decoding and enhancement, and a video super-resolution module.

[0039] The intelligent stream encoder takes the original video image (2160P) as input and encodes it into an intelligent stream (1080P) with an extremely low bitrate.

[0040] The intelligent stream decoder decodes and enhances extremely low bitrate intelligent streams (1080P) into high-quality video images and outputs enhanced video images (2160P).

[0041] The ROI (Region of Interest) identification and encoding process imports user-configured categories of interest, uses a low-load AI model to detect ROI regions, segments the ROI images from the original image, and compresses the information of the ROI images before encoding them into the SEI (Search Engine Identifier) ​​auxiliary stream. The intelligent bitstream encoder and ROI identification and encoding can be configured for serial or parallel task execution. Leveraging the characteristics of heterogeneous multi-core processors, parallel execution offers performance and latency advantages. The ROI detection module is an external module of the intelligent encoding system, with built-in face ROI detection and pickup, and allows users to define and implement their own ROI detection and pickup, replacing the built-in ROI module through an interface. This intelligent encoding ROI detection module is not a general ROI encoder, but rather a ROI encoder specifically designed for intelligent encoding; that is, ROI encoding and decoding focus on the logic of intelligent encoding, including domain-specific processing procedures.

[0042] ROI region decoding and enhancement involves parsing the SEI syntax to analyze the ROI region and image. The ROI image is then overlaid onto the super-resolution video image to form an enhanced video image (2160P). The SEI bitstream of the intelligently encoded ROI region (relative to the SEI bitstream of a general ROI) has the following special characteristics: 1) The SEI bitstream contains the ROI region used for intelligent encoding and the enhanced SEI syntax elements suitable for intelligent encoding. 2) There is a special mapping relationship between the intelligently encoded ROI region and the enhanced SEI syntax elements used for intelligent encoding, requiring the decoder to parse, process, and enhance the ROI image according to the syntax elements.

[0043] The video super-resolution module preferably uses a 2D hardware image processor or an AI model super-resolution algorithm. It restores 1080P video images to 2160P. 2D hardware image processors have lower system resource consumption and offer moderate quality. AI model super-resolution algorithms have higher system resource consumption and offer better quality. The video super-resolution module is selected based on system load or user configuration. Following the video super-resolution module is a post-processing module for overlay and filtering. The filtered Region of Interest (ROI) image is overlaid onto the super-resolution image according to the ROI information.

[0044] Figure 2 This is a flowchart illustrating an encoding method according to an embodiment of the present disclosure. Figure 2 As shown, the encoding method provided in this embodiment includes the following steps S11 to S15.

[0045] Step S11: Acquire video images.

[0046] In some embodiments, video images are acquired from a player, transcoder, display system, or camera. The video images may include all images in the video.

[0047] In some embodiments, the video image is obtained by directly transferring the video image via shared memory.

[0048] Step S12: Detect the ROI of the video image based on the ROI detection model to obtain the ROI information of the video image. The ROI information of the video image includes ROI location information and ROI category.

[0049] In some embodiments, the ROI location information includes ROI size and ROI coordinates, where size can refer to width and height, and coordinates can refer to x-axis coordinates and y-axis coordinates.

[0050] In some embodiments, a method for detecting the ROI of a video image based on a ROI detection model to obtain the ROI information of the video image includes: importing user-configured categories of interest; and detecting the ROI of the video image based on the ROI detection model and the categories of interest to obtain the ROI information of the video image. The categories of interest may be, for example, license plates, faces, or text.

[0051] In some embodiments, the ROI detection model can be a lightweight YOLO model. YOLO is a deep learning-based object detection algorithm primarily used for real-time object identification in images or videos.

[0052] Step S13: Extract the ROI of the video image based on the ROI information of the video image to obtain the ROI image.

[0053] In some embodiments, the ROI image can be an image stitched together from several ROIs extracted from the video image. Specifically, it can be based on the ROI information, extracting the corresponding ROI region (e.g., face, license plate, original pixels rather than categories), and stitching each ROI together to create a new image (which can be called an added layer image, the size of which can be flexibly configured according to the actual situation, the default is 1024*1080).

[0054] Step S14: Encode the ROI image to obtain the ROI image encoding result.

[0055] In some embodiments, the method for encoding the ROI image to obtain the ROI image encoding result includes: sorting the ROI image using a similarity matching algorithm to obtain sorted ROI images; and encoding the sorted ROI images to obtain the ROI image encoding result. Sorting the ROI objects can improve the compression ratio of the encoding.

[0056] In some embodiments, steps S12 to S14 can be configured to execute serial or parallel tasks. By utilizing the characteristics of heterogeneous multi-core processors, parallel execution has advantages in performance and latency.

[0057] Step S15: Pack the ROI information of the video image and the ROI image encoding result into the SEI auxiliary stream packet.

[0058] In some embodiments, the method for packaging the ROI information of the video image and the ROI image encoding result into an SEI auxiliary stream packet includes: dividing the ROI data into blocks and transmitting the ROI data through multiple SEI frames. The ROI data may include the ROI information of the video image and the ROI image encoding result.

[0059] As described above, the ROI of the video image is detected using a ROI detection model to obtain ROI information. Based on this ROI information, the ROIs of the video image are extracted to obtain an ROI image. The ROI image is then encoded to obtain an ROI image encoding result. Furthermore, the ROI information and the ROI image encoding result are packaged into a Supplemental Enhancement Information (SEI) auxiliary stream packet.

[0060] In the encoding method, by detecting and extracting the ROI of the video image, further encoding the ROI image, and packaging the ROI information and the ROI image encoding result into the SEI auxiliary stream packet, the encoding effect of the ROI region can be preserved to the greatest extent, rather than just saving the ROI category, thereby improving the encoding effect of the ROI region of the video image.

[0061] Figure 3 This is a flowchart illustrating the process of detecting the Region of Interest (ROI) of a video image using a Region of Interest (ROI) detection model according to an embodiment of this disclosure to obtain ROI information of the video image. Figure 3 As shown, obtaining the ROI information of the video image includes the following steps S21 to S23.

[0062] Step S21: Load the ROI detection model and perform detection on the video image based on the ROI detection model to obtain the ROI information of the video image.

[0063] In some embodiments, the ROI detection model can be loaded and used as a peripheral module of the encoder.

[0064] Step S22: When the ROI associated with the ROI information of the previous several frames of the video image disappears in the ROI information of the video image, motion estimation processing is performed on the video image based on the ROI information of the previous several frames of the video image to obtain the estimated ROI position information of the disappeared ROI in the video image.

[0065] In some embodiments, the vanishing ROI may refer to an ROI that is partially occluded or missed during motion. The vanishing ROI of the video image can be determined by the detection process and ROI information from several frames preceding the video image. For example, if an ROI appears in the first 5 frames but disappears in the 6th frame (the current video image), then that ROI is a vanishing ROI.

[0066] Step S23: Update the estimated ROI location information and the category of the disappeared ROI to the ROI information of the video image.

[0067] In some embodiments, updating the estimated ROI location information and the category of the vanished ROI to the ROI information of the video image may refer to filling the estimated ROI location information and the category of the vanished ROI into the ROI information of the video image.

[0068] Figure 4 This is a flowchart illustrating the process of detecting the Region of Interest (ROI) of a video image using a Region of Interest (ROI) detection model according to an embodiment of this disclosure to obtain ROI information of the video image. Figure 4 As shown, obtaining the ROI information of the video image includes the following steps S31 to S34.

[0069] Step S31: Based on the ROI size in the ROI information of the video image, filter the ROI information of the video image to obtain the filtered ROI information of the video image.

[0070] In some embodiments, the filtered ROI information of the video image may refer to the ROI information after filtering out the relevant information of ROIs that are too large or too small in the ROI information of the video image. The specific filtering size can be flexibly set according to the actual situation, and this embodiment does not explicitly limit it.

[0071] Step S32: Align the ROI coordinates in the filtered ROI information to obtain aligned ROI information.

[0072] In some embodiments, aligning the ROI coordinates in the filtered ROI information to obtain aligned ROI information includes aligning the ROI coordinates and ROI dimensions (width and height) in the filtered ROI information to obtain aligned ROI information. In this embodiment, the alignment of both the ROI dimensions and ROI coordinates can be done with a 16-axis alignment.

[0073] Step S33: Based on the aligned ROI information and the ROI information of the previous frame of the video image, obtain the structural similarity index and absolute difference of the ROIs of the same category as those in the ROIs of the previous frame of the video image in the aligned ROI information, and the ROIs in the ROI information of the previous frame of the video image have their corresponding sequential positions.

[0074] In some embodiments, the calculation of the Structural Similarity Index (SSIM) and the Sum of Absolute Differences (SAD) will not be described again in this embodiment. In the ROI information of a frame in the video image, each ROI has a corresponding sequential position; for example, ROI1 can correspond to position 1, ROI2 can correspond to position 2, and so on.

[0075] Step S34: Based on the structural similarity index and the sum of absolute differences, sort the ROIs in the aligned ROI information to obtain the order position of the ROIs in the aligned ROI information.

[0076] In some embodiments, sorting ROIs in the aligned ROI information based on the structural similarity index and the sum of absolute differences includes: determining the correlation between the ROIs in the aligned ROI information and the ROIs in the previous frame of the video image based on the structural similarity index and the sum of absolute differences; and if there are two highly correlated ROIs in the video image and the previous frame of the video image, then the two highly correlated ROIs are sorted in the same order; otherwise, based on the ROI position information in the aligned ROI information, the ROIs with low correlation are sorted.

[0077] Sorting by the same positional order means that the positional order of ROIs in the video image is the same as the positional order of ROIs in the previous frame of the video image. For example, if the positional order of ROI1 in the previous frame of the video image is 1, and ROI4 in the video image has a high similarity to ROI1, then ROI4 is also sorted by its position 1. ROIs with low relevance can refer to the remaining ROIs in the aligned ROI information after excluding the ROIs with high relevance. This can include ROIs with low relevance calculated based on SSIM and SAD, and ROIs newly added to the video image compared to the previous frame. The level of relevance can be flexibly set according to actual conditions, and this embodiment does not explicitly limit this.

[0078] In some embodiments, sorting ROIs with low relevance in the aligned ROI information based on ROI position information includes: sorting ROIs with low relevance in the aligned ROI information based on ROI size to obtain a first sorted ROI; and sorting the first sorted ROIs based on ROI coordinates to obtain a second sorted ROI, wherein the order position of the second sorted ROI is the order position of the ROI in the aligned ROI information.

[0079] In some embodiments, the first sorted ROIs are ordered in descending order of their size, and the second sorted ROIs are ordered first in ascending order of their y-coordinates, and then in ascending order of their x-coordinates.

[0080] Step S35: Based on the sequential position of the ROI in the aligned ROI information, the ROI in the aligned ROI information is spliced ​​together to obtain spliced ​​ROI information.

[0081] In some embodiments, the spliced ​​ROI information may refer to the information obtained by splicing the corresponding ROI information according to the order position of the ROI in the aligned ROI information.

[0082] Figure 5 This is a flowchart illustrating an encoding method according to an embodiment of the present disclosure. Figure 5 As shown, the encoding method includes the following steps S41 to S42.

[0083] Step S41: The video image is downsampled to obtain a low-resolution video image.

[0084] Step S42: Encode the low-resolution video image using a hardware encoder to obtain the bitstream of the low-resolution video image.

[0085] In some embodiments, the method for encoding the low-resolution video image based on a hardware encoder to obtain the bitstream of the low-resolution video image includes: allocating a medium bitrate to the dynamic region of the video image and a low bitrate to the static region of the video image during the encoding process.

[0086] Step S43: Package the bitstream and the SEI auxiliary stream packet into a predetermined transmission format.

[0087] In some embodiments, the predetermined transmission format can be flexibly set according to actual conditions, and this embodiment does not explicitly limit it.

[0088] Figure 6 This is a flowchart illustrating how, according to an embodiment of the present disclosure, the ROI of a video image is extracted to obtain an ROI image based on the ROI information of the video image. Figure 6 As shown, obtaining the ROI image includes the following steps S51 and S52.

[0089] Step S51: Define ROI level, which is used to assign pixel affiliation to overlapping areas in the ROI region according to the ROI level.

[0090] In some embodiments, assigning pixel ownership to overlapping regions in an ROI region according to the ROI level can mean that the pixel ownership of overlapping regions in an ROI region is assigned to the one with the higher ROI level. For example, if the face level is Lv1 and the text level is Lv2, and Lv1 > Lv2, the pixel is assigned to the face.

[0091] Step S52: Based on the ROI level and the ROI information of the video image, perform ROI extraction processing on the video image to obtain the ROI image.

[0092] Figure 7 This is a flowchart illustrating a decoding method according to an embodiment of the present disclosure. Figure 7 As shown, the decoding method includes the following steps S61 to S64.

[0093] Step S61: Obtain the SEI auxiliary stream packet and the bitstream of the video image.

[0094] In some embodiments, the bitstream can be a low-resolution video image bitstream, or a standard H.264 / H.265 bitstream, which can be decoded using a chip hardware decoder.

[0095] Step S62: Decode the SEI auxiliary stream packet to obtain the ROI information and ROI image of the video image.

[0096] In some embodiments, the method for decoding the SEI auxiliary stream packet to obtain the ROI information and ROI image of the video image includes: parsing the SEI auxiliary stream packet according to the SEI syntax elements in the SEI auxiliary stream packet to obtain the ROI information and ROI image of the video image.

[0097] In some embodiments, the method for decoding the SEI auxiliary stream packet to obtain the ROI information and ROI image of the video image includes: when the ROI location information in the SEI auxiliary stream packet is lost, the ROI location information can be predicted by motion estimation based on the ROI data. This embodiment will not elaborate further on this.

[0098] Step S63: Decode the bitstream to obtain the video image.

[0099] Step S64: Generate an enhanced video image based on the ROI information, the ROI image, and the video image.

[0100] In some embodiments, the enhanced video image may refer to a high-quality image that enhances the video image.

[0101] In some embodiments, the decoding method may further include: reconstructing the video image, which is a low-resolution video image, using an AI super-resolution model to obtain a high-resolution video image. The AI ​​super-resolution model may be an RGA (Residual Global Attention) super-resolution model or an ECBSR model, etc., and can be flexibly selected according to the chip's capabilities; this embodiment does not explicitly limit this. The ECBSR model is a lightweight real-time super-resolution model designed for mobile devices, whose core is to achieve efficient processing through reparameterizable edge-guided convolutional blocks.

[0102] Figure 8 This is a flowchart illustrating the generation of an enhanced video image based on the ROI information, the ROI image, and the video image according to an embodiment of this disclosure. For example... Figure 8 As shown, generating an enhanced video image includes the following steps S71 and S72.

[0103] Step S71: Map the ROI to the matching position of the high-resolution video image based on the ROI information.

[0104] Step S72: Based on the matching position, the ROI image is mapped onto the high-resolution video image with a gradient edge to generate the enhanced video image.

[0105] In some embodiments, the gradient edge method may refer to applying an alpha gradient blend that changes linearly with distance within a 5-8 pixel range of the ROI boundary. Edge gradient blending eliminates seams and makes image transitions more natural.

[0106] In one embodiment of this disclosure, bilateral filtering and mean filtering are performed on the high-frequency and low-frequency components of the ROI boundary in the ROI image or enhanced video image, respectively, to obtain a filtered image. The high-frequency component may be, for example, the texture of the boundary. The low-frequency component may be, for example, skin color.

[0107] In one embodiment of this disclosure, the workflow of the intelligent bitstream encoder includes the following steps: (1) acquiring video images; (2) downsampling to low resolution using hardware acceleration; (3) encoding the low-resolution bitstream using a hardware encoder; (4) performing ROI identification using a model; (5) encoding information such as ROI images into an SEI auxiliary stream; and (6) packaging the video bitstream and the SEI auxiliary stream into a transmission format.

[0108] Step (1) Acquire video images. The front-end of the intelligent stream encoder may be: player, transcoder, display system, or camera. High-resolution image frame buffer bandwidth is under great pressure, so it is preferable for the intelligent stream encoder to directly transfer video images through shared memory to avoid CPU copying.

[0109] Step (2) uses hardware acceleration to downsample to a lower resolution. Image scaling consumes a lot of computational and memory resources, so it is preferable to use image processing hardware supported by the system for downsampling.

[0110] Step (3) Encodes the low-resolution bitstream using a hardware encoder. Hardware encoder parameter tuning is limited and it's difficult to dynamically adapt to the ROI, so a technique for adjusting the encoder ROI is used. The low-resolution bitstream is judged based on whether it's static or dynamic; a medium bitrate is allocated to dynamic regions, and a lower bitrate is allocated to static and dynamic regions.

[0111] Step (4) Use the model for ROI identification and segmentation. Import the user-configured categories of interest and use a two-level ROI detection model to detect regions of the categories of interest (license plates, faces, or text, etc.). A two-level ROI detection architecture is adopted, using a lightweight YOLO model to quickly locate ROI candidate regions and combining optical flow tracking of ROI cross-frame motion. Temporal correlation is used to reduce the amount of computation per frame. For example, full detection is performed every N frames, and motion estimation is used for intermediate frames.

[0112] Step (5) encodes ROI image and other information into the SEI auxiliary stream. SEI has limited capacity, so ROI data needs to be compressed efficiently.

[0113] Step (6) packages the video stream and SEI auxiliary stream into a transmission format. Ensure that the ROI data is strictly aligned with the video frames.

[0114] In one embodiment of this disclosure, the workflow of the intelligent stream decoder includes the following steps: (1) decoding a low-resolution video stream; (2) reconstructing a high-resolution image from the low-resolution stream using a model; (3) decoding the ROI information of the SEI auxiliary stream; (4) reconstructing the ROI region image using the ROI information; (5) mitigating the boundary effect of the ROI using ROI filtering; and (6) rendering the decoded image of the intelligent stream onto a display device.

[0115] Step (1) Decode the low-resolution video stream. The low-resolution video stream is a standard H.264 / H.265 stream, which can be decoded using a chip hardware decoder.

[0116] Step (2) Use the model to reconstruct the high-resolution image from the low resolution. Select the super-resolution module according to the chip's capabilities; RK3568 can use RGA super-resolution; RK3576 and RK3588 can use AI model super-resolution algorithm. The AI ​​model super-resolution algorithm can use a simplified version of the ECBSR model. The AI ​​model super-resolution algorithm can solve the line breakage problem caused by downsampling and reduce image jaggedness.

[0117] Step (3) decodes the ROI information of the SEI auxiliary stream. When an ROI is lost, its location is estimated by motion. The ROI information includes the ROI image, its location information, and its category. Steps 1 and 3) can be configured to run serially or in parallel. Taking advantage of the characteristics of heterogeneous multi-core processors, parallel operation has advantages in performance and latency.

[0118] Step (4) Reconstruct the ROI region image using ROI information. Map the ROI region to the matching location in the high-resolution image, and apply the ROI image to the high-resolution image with a gradient edge. Apply an alpha gradient blend that varies linearly with distance within a 5-8 pixel range of the ROI boundary. The edge gradient blending eliminates seams and makes the image transition more natural.

[0119] Step (5) uses ROI filtering to mitigate the boundary effects of ROIs. Bilateral filtering is used for high-frequency details (such as texture), and mean filtering is used for low-frequency components (such as skin color).

[0120] In one embodiment of this disclosure, the intelligent stream ROI identification and encoding method includes the following steps: (1) loading the ROI identification model; (2) calling the AI ​​engine to identify ROI information; (3) segmenting the ROI region image from the high-resolution image; (4) integrating the ROI region and compressing and encoding it; (5) packaging the ROI information and image into the SEI auxiliary stream packet; and (6) outputting it to the subsequent module.

[0121] Step (1) Load the ROI recognition model. Import the user-configured categories of interest and use a two-level ROI detection model (YOLO model + UNet model) to detect regions of interest (license plates, faces, or text, etc.).

[0122] Step (2) invokes the AI ​​engine to identify ROI information. A two-level ROI detection architecture is adopted, using a lightweight YOLO model to quickly locate ROI candidate regions and combining optical flow tracking of ROI cross-frame motion. Temporal correlation is utilized to reduce frame-by-frame computation. For example, full detection is performed every 5 frames, and motion estimation is used in intermediate frames to filter, align, sort, and stitch ROI regions. Areas with large ROI areas are filtered out, and then SSIM and SAD values ​​are calculated and sorted (proper arrangement can improve compression ratio).

[0123] Step (3) Extract the ROI region image from the high-resolution image. The coordinates of the ROI are directly segmented, and the coordinate information of the ROI region is aligned. Define the ROI level (e.g., face Lv1 > text Lv2), and assign pixels to overlapping areas according to priority.

[0124] Step (4) Compress the ROI region image. Perform high bitrate video encoding on the ROI region.

[0125] Step (5) Pack the ROI information and image into the SEI auxiliary stream packet. The SEI data is segmented and encoded, and the ROI data is divided into blocks and transmitted through multiple SEI frames. The decoder then reassembles the data.

[0126] Step (6) outputs the ROI region image to the subsequent modules. The subsequent modules may include: a player, a smart stream decoder, a video server, a file server, etc.

[0127] In one embodiment of this disclosure, the intelligent stream ROI decoding and filtering method includes the following steps: (1) acquiring SEI auxiliary stream packets; (2) parsing ROI information and ROI images from the SEI; (3) ROI filtering processing; (4) rendering the ROI image to the ROI region of a high-resolution image; and (5) other processing by subsequent modules.

[0128] Step (1) Obtain the SEI auxiliary flow packet.

[0129] Step (2) parses ROI information and ROI images from SEI. The decoding end needs to decode the ROI images via video. The ROI information includes a mapping relationship between low-resolution and high-resolution resolutions.

[0130] Step (3) ROI filtering. Filter the pixels on both sides of the boundary with a certain intensity.

[0131] Step (4) Render the ROI image onto the high-resolution ROI region. Apply an alpha gradient blend that varies linearly with distance within a 5-8 pixel range around the ROI boundary. This edge gradient blending eliminates seams and makes the image transition more natural.

[0132] Step (5) Further processing by subsequent modules. Subsequent modules may include: players, display devices, etc.

[0133] The encoding process of this embodiment, at a resolution of 3840*2160 (taking 4K resolution as an example) and above, involves identifying the corresponding ROI region (e.g., face, license plate, original pixels rather than categories) through ROI recognition, stitching together each ROI to create a new image (called the enhancement layer image, the size of which can be customized, the default is 1024*1080), adjusting the arrangement order of the ROIs through a similarity matching algorithm (to improve the subsequent encoding compression rate), and then encoding the enhancement layer image. The encoded bitstream information is stored in the SEI of the base layer bitstream for easy parsing by the decoding end.

[0134] Figure 9 This is a schematic diagram illustrating the structure of the encoding device according to an embodiment of the present disclosure. Figure 9 As shown, the encoding device includes: a video image acquisition module 910, a video image detection module 920, an ROI image extraction module 930, an ROI image encoding module 940, an ROI information packaging module 950, and a bitstream packaging module 960.

[0135] The video image acquisition module 910 is configured to acquire video images.

[0136] The video image detection module 920 is configured to detect the ROI of the video image based on the region of interest (ROI) detection model to obtain the ROI information of the video image, wherein the ROI information of the video image includes ROI location information and ROI category.

[0137] The ROI image extraction module 930 is configured to extract the ROI of the video image based on the ROI information of the video image to obtain an ROI image.

[0138] The ROI image encoding module 940 is configured to encode the ROI image to obtain the ROI image encoding result.

[0139] The ROI information packaging module 950 is configured to package the ROI information of the video image and the ROI image encoding result into the SEI auxiliary stream packet.

[0140] The stream packaging module 960 is configured to package the stream based on the video image and the SEI auxiliary stream packet into a predetermined transmission format.

[0141] The video image acquisition module 910, video image detection module 920, ROI image extraction module 930, ROI image encoding module 940, and ROI information packaging module 950 provided in this embodiment are... Figure 2 Steps S11 to S15 of the encoding method, or the detailed steps or actions in its implementation method, correspond one-to-one with the bitstream packaging module 960. Figure 5 Step S43 corresponds one-to-one, and will not be repeated here.

[0142] Figure 10 This is a schematic diagram illustrating the structure of a decoding device according to an embodiment of the present disclosure. Figure 10 As shown, the decoding device includes: a bitstream acquisition module 1010, an auxiliary stream packet parsing module 1020, a bitstream parsing module 1030, and an image generation module 1040.

[0143] The bitstream acquisition module 1010 is configured to acquire the bitstream of SEI auxiliary stream packets and video images.

[0144] The auxiliary stream packet parsing module 1020 is configured to decode the SEI auxiliary stream packet to obtain the ROI information and ROI image of the video image.

[0145] The bitstream parsing module 1030 is configured to decode the bitstream to obtain the video image.

[0146] The image generation module 1040 is configured to generate an enhanced video image based on the ROI information, the ROI image, and the video image.

[0147] The code stream acquisition module 1010, auxiliary stream packet parsing module 1020, code stream parsing module 1030, and image generation module 1040 provided in this embodiment are... Figure 7 The steps S61 to S64 of the decoding method or the detailed steps or actions in its implementation method correspond one-to-one, and will not be repeated here.

[0148] In the several embodiments provided in this disclosure, it should be understood that the disclosed systems, apparatuses, or methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules / units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or units may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of apparatuses or modules or units may be electrical, mechanical, or other forms.

[0149] The modules / units described as separate components may or may not be physically separate. The components shown as modules / units may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules / units can be selected to achieve the objectives of the embodiments of this disclosure, depending on actual needs. For example, the functional modules / units in the various embodiments of this disclosure may be integrated into one processing module, or each module / unit may exist physically separately, or two or more modules / units may be integrated into one module / unit.

[0150] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0151] This disclosure also provides a computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the encoding method and apparatus, decoding method and apparatus, electronic device, and storage medium provided in this disclosure. Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing a processor. The program can be stored in a computer-readable storage medium, which is a non-transitory medium, such as random access memory, read-only memory, flash memory, hard disk, solid-state hard disk, magnetic tape, floppy disk, optical disk, and any combination thereof. The storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0152] This disclosure also provides an electronic device including a memory and a processor. The memory is configured to store information associated with an image. The processor is electrically coupled to the memory and configured to perform the encoding and / or decoding methods provided in this disclosure.

[0153] In some implementations, the memory may include a computer system readable medium in the form of volatile memory, such as RAM and / or cache memory. The electronic device may further include other removable / non-removable, volatile / non-volatile computer system storage media. The memory may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this disclosure.

[0154] In some implementations, the processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc. In other implementations, the processor can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0155] In summary, the embodiments of this disclosure provide encoding methods and apparatus, decoding methods and apparatus, electronic devices, and storage media. By identifying and extracting the ROI of the video image, further encoding the ROI image, and packaging the ROI information and the ROI image encoding result into the SEI auxiliary stream packet, the encoding effect of the ROI region can be preserved to the greatest extent, rather than just saving the ROI category, thereby improving the encoding effect of the ROI region of the video image.

[0156] The above embodiments are merely illustrative of the principles and effects of this disclosure and are not intended to limit this disclosure. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this disclosure. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this disclosure should still be covered by the claims of this disclosure.

Claims

1. An encoding method, characterized in that, include: Acquire video images; The ROI of the video image is detected based on the Region of Interest (ROI) detection model to obtain the ROI information of the video image. The ROI information of the video image includes ROI location information and ROI category. The ROI of the video image is extracted based on the ROI information of the video image to obtain the ROI image; The ROI image is encoded to obtain the ROI image encoding result; as well as The ROI information of the video image and the ROI image encoding result are packaged into the supplementary enhancement information (SEI) auxiliary stream packet.

2. The encoding method according to claim 1, characterized in that, The ROI detection of the video image based on the Region of Interest (ROI) detection model to obtain the ROI information of the video image includes: Load the ROI detection model and detect the video image based on the ROI detection model to obtain the ROI information of the video image; When a ROI associated with ROI information from several previous frames of the video image disappears from the ROI information of the video image, motion estimation processing is performed on the video image based on the ROI information from the several previous frames to obtain the estimated ROI location information of the disappeared ROI in the video image; and The estimated ROI location information and the category of the vanished ROI are updated to the ROI information of the video image.

3. The encoding method according to claim 2, characterized in that, The ROI location information includes ROI size and ROI coordinates, and the ROI detection of the video image based on the Region of Interest (ROI) detection model to obtain the ROI information of the video image also includes: Based on the ROI size in the ROI information of the video image, the ROI information of the video image is filtered to obtain the filtered ROI information of the video image; Align the ROI coordinates in the filtered ROI information to obtain aligned ROI information; Based on the aligned ROI information and the ROI information of the previous frame of the video image, the structural similarity index and absolute difference of the ROIs of the same category as those in the ROIs of the previous frame of the video image are obtained in the aligned ROI information. The ROIs in the ROI information of the previous frame of the video image have their corresponding sequential positions. Based on the structural similarity index and the sum of absolute differences, the ROIs in the aligned ROI information are sorted to obtain the sequential positions of the ROIs in the aligned ROI information; and Based on the sequential position of the ROIs in the aligned ROI information, the ROIs in the aligned ROI information are spliced ​​together to obtain spliced ​​ROI information.

4. The encoding method according to claim 3, characterized in that, Based on the structural similarity index and the sum of absolute differences, the sorting process for ROIs in the aligned ROI information includes: Based on the structural similarity index and the absolute difference, the correlation between the ROI in the aligned ROI information and the ROI in the previous frame of the video image is determined; and If there are two highly correlated ROIs in the video image and the previous frame of the video image, the two highly correlated ROIs are sorted in the same order; otherwise, based on the ROI position information in the aligned ROI information, the ROIs with low correlation are sorted.

5. The encoding method according to claim 4, characterized in that, Based on the ROI location information in the aligned ROI information, sorting the ROIs with low relevance in the aligned ROI information includes: Based on the ROI size in the aligned ROI information, the ROIs with low relevance in the aligned ROI information are sorted to obtain the first sorted ROIs; and Based on the ROI coordinates corresponding to the first sorted ROI, the first sorted ROI is sorted to obtain the second sorted ROI, and the order position of the second sorted ROI is the order position of the ROI in the aligned ROI information.

6. The encoding method according to claim 1, characterized in that, Also includes: The video image is downsampled to obtain a low-resolution video image; The low-resolution video image is encoded to obtain the bitstream of the low-resolution video image; as well as The bitstream and the SEI auxiliary stream packet are packaged into a predetermined transmission format.

7. The encoding method according to claim 1, characterized in that, Extracting the ROI from the video image based on its ROI information to obtain an ROI image includes: Define ROI levels, which are used to assign pixel affiliations to overlapping regions within an ROI according to the ROI level; and Based on the ROI level and the ROI information of the video image, the ROI of the video image is extracted to obtain the ROI image.

8. A decoding method, characterized in that, include: Obtain the SEI auxiliary stream packets and video image bitstream; Decode the SEI auxiliary stream packet to obtain the ROI information and ROI image of the video image; Decode the bitstream to obtain the video image; as well as An enhanced video image is generated based on the ROI information, the ROI image, and the video image.

9. The decoding method according to claim 8, characterized in that, Also includes: The video image, which is a low-resolution video image, is reconstructed using an AI super-resolution model to obtain a high-resolution video image.

10. The decoding method according to claim 9, characterized in that, Generating an enhanced video image based on the ROI information, the ROI image, and the video image includes: Based on the ROI information, the ROI is mapped to the matching position in the high-resolution video image; and The ROI image is mapped onto the high-resolution video image with a gradient edge based on the matching location to generate the enhanced video image.

11. The decoding method according to claim 8, characterized in that, The decoding method further includes: Bilateral filtering and mean filtering are performed on the high-frequency and low-frequency components of the ROI boundary in the ROI image or the enhanced video image, respectively, to obtain the filtered image.

12. An encoding device, characterized in that, include: The video image acquisition module is configured to acquire video images; The video image detection module is configured to detect the Region of Interest (ROI) of the video image based on the Region of Interest (ROI) detection model to obtain the ROI information of the video image, wherein the ROI information of the video image includes ROI location information and ROI category; The ROI image extraction module is configured to extract the ROI of the video image based on the ROI information of the video image to obtain the ROI image; The ROI image encoding module is configured to encode the ROI image to obtain the ROI image encoding result; The ROI information packaging module is configured to package the ROI information of the video image and the ROI image encoding result into the SEI auxiliary stream packet; as well as The bitstream packaging module is configured to package the bitstream based on the video image and the SEI auxiliary stream packet into a predetermined transmission format.

13. A decoding device, characterized in that, include: The bitstream acquisition module is configured to acquire the bitstreams of SEI auxiliary stream packets and video images; The auxiliary stream packet parsing module is configured to decode the SEI auxiliary stream packet to obtain the ROI information and ROI image of the video image; The bitstream parsing module is configured to decode the bitstream to obtain the video image; as well as The image generation module is configured to generate an enhanced video image based on the ROI information, the ROI image, and the video image.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the encoding method according to any one of claims 1 to 7 and / or the decoding method according to any one of claims 8 to 11.

15. An electronic device, characterized in that, include: The memory is configured to store information associated with the image; as well as A processor, electrically coupled to the memory, is configured to perform the encoding method according to any one of claims 1 to 7 and / or the decoding method according to any one of claims 8 to 11.

Images