Video encoding and decoding method, device, computer device and storage medium

Abstract

The application discloses a video coding and decoding method, device, computer equipment and storage medium, wherein the video coding method can include dividing a video frame image to be coded to obtain at least two block images; identifying a region of interest in the block image; determining a sub-scaling ratio of the block image based on a target scaling ratio of the video frame image and attribute information of the region of interest; and coding the block image based on the sub-scaling ratio of the block image to obtain a coded image of the video frame image. The embodiment of the application can code different blocks of a video frame image based on different scaling ratios, can avoid image detail loss of the region of interest, and is beneficial to improving the display effect of the image.

Description

Technical Field

[0001] This application relates to the field of image processing technology, specifically to a video encoding and decoding method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the continuous development of network, multimedia, and hardware technologies, video surveillance, as one of the important video applications, has been widely used in fields such as security, intelligent security, and smart homes. In video surveillance systems, the raw surveillance video data is enormous; only after video encoding processing can the surveillance video data be effectively transmitted and stored.

[0003] Currently, the Versatile Video Coding (VVC) standard employs Adaptive Resolution Change (ARC) technology. ARC allows video sequences to be transmitted at different resolutions depending on network conditions. For example, when network bandwidth is low, low-resolution video is transmitted; when bandwidth is high, the original resolution video is transmitted, thereby reducing the bitrate. However, changing the resolution can cause a loss of image details, especially in the region of interest. Summary of the Invention

[0004] This application provides a video encoding and decoding method, apparatus, computer device, and storage medium that can encode different blocks of a video frame image based on different scaling ratios, thereby avoiding the loss of image details in the region of interest and improving the image display effect.

[0005] In a first aspect, embodiments of this application provide a video encoding method, wherein:

[0006] The video frame image to be encoded is divided into at least two block images;

[0007] Identify the region of interest in the block image;

[0008] The sub-scaling ratio of the block image is determined based on the target scaling ratio of the video frame image and the attribute information of the region of interest;

[0009] The block image is encoded based on the sub-scaling ratio of the block image to obtain the encoded image of the video frame image;

[0010] The mode information of the encoded image and the block image is sent to the decoding end, wherein the mode information of the block image includes at least the sub-scaling ratio of the block image.

[0011] In one possible example, determining the sub-scaling ratio of the block image based on the target scaling ratio of the video frame image and the attribute information of the region of interest includes:

[0012] The interest value of the block image is calculated based on the attribute information of the region of interest, wherein the attribute information includes at least one of the following: area size, object type, and relative position;

[0013] The sub-scaling ratio of the block image is determined based on the interest value of the block image and the target scaling ratio of the video frame image, wherein the larger the interest value of the block image, the smaller the sub-scaling ratio of the block image, and the weighted average of the sub-scaling ratios is equal to the target scaling ratio.

[0014] In one possible example, if the areas of the block images are equal, then the preset weights of the sub-scaling ratios are equal; if the areas of the block images are unequal, then the preset weights of the sub-scaling ratios are determined by the attribute information of the block images.

[0015] In one possible example, the video frame image to be encoded is divided into at least two block images, including:

[0016] The size of the block image is determined based on the resolution of the video frame image to be encoded;

[0017] The video frame image is divided based on the size to obtain at least two block images.

[0018] In one possible example, encoding the block image based on the sub-scaling ratio of the block image to obtain the encoded image of the video frame image includes:

[0019] Obtain the binary data corresponding to the block image;

[0020] A context model is obtained by modeling based on the context information of the block image;

[0021] Based on the sub-scaling ratio of the block image and the context model, binary arithmetic encoding is performed on the binary data to obtain the encoded image of the video frame image.

[0022] Secondly, embodiments of this application provide a video decoding method, wherein:

[0023] The encoded image of the video frame image and the mode information of each block image in the video frame image are received from the encoding end. The mode information of the block image includes at least the sub-scaling ratio of the block image.

[0024] The encoded image is decoded based on the pattern information to obtain the video frame image.

[0025] Thirdly, embodiments of this application provide a video encoding apparatus, wherein:

[0026] The partitioning unit is used to divide the video frame image to be encoded into at least two block images;

[0027] The recognition unit is used to identify the region of interest in the block image;

[0028] The determining unit is used to determine the sub-scaling ratio of the block image based on the target scaling ratio of the video frame image and the attribute information of the region of interest;

[0029] The encoding unit is used to encode the block image based on the sub-scaling ratio of the block image to obtain the encoded image of the video frame image;

[0030] The sending unit sends the mode information of the encoded image and the block image to the decoding end, wherein the mode information of the block image includes at least the sub-scaling ratio of the block image.

[0031] In one possible example, the determining unit is specifically used to calculate the interest value of the block image based on the attribute information of the region of interest, wherein the attribute information includes at least one of the following: area size, object type, and relative position; and to determine the sub-scaling ratio of the block image based on the interest value of the block image and the target scaling ratio of the video frame image, wherein the larger the interest value of the block image, the smaller the sub-scaling ratio of the block image.

[0032] In one possible example, the weighted average of the sub-scaling ratios is equal to the target scaling ratio, wherein if the areas of the block images are equal, the preset weights of the sub-scaling ratios are equal; if the areas of the block images are unequal, the preset weights of the sub-scaling ratios are determined by the attribute information of the block images.

[0033] In one possible example, the partitioning unit is specifically used to determine the size of the block image based on the resolution of the video frame image to be encoded; and to partition the video frame image based on the size to obtain at least two block images.

[0034] In one possible example, the encoding unit is specifically used to acquire binary data corresponding to the block image; to model based on the context information of the block image to obtain a context model; and to perform binary arithmetic encoding on the binary data based on the sub-scaling ratio of the block image and the context model to obtain the encoded image of the video frame image.

[0035] Fourthly, embodiments of this application provide a video decoding apparatus, wherein:

[0036] The receiving unit is used to receive the encoded image of the video frame image and the mode information of each block image in the video frame image from the encoding end. The mode information of the block image includes at least the sub-scaling ratio of the block image.

[0037] The decoding unit is used to decode the encoded image based on the mode information to obtain the video frame image.

[0038] Fifthly, embodiments of this application provide a computer device including a processor, a memory, and a communication interface, wherein the memory stores a computer program configured to be executed by the processor, and the computer program includes methods for performing the methods described in the first or second aspect.

[0039] Sixthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, the computer program causing a computer device to execute and implement the method as described in the first or second aspect.

[0040] In a seventh aspect, embodiments of this application provide a combined processing apparatus, characterized in that the combined processing apparatus includes the apparatus as described in the third or fourth aspect, a general interconnect interface, and other processing apparatuses; the apparatus interacts with the other processing apparatuses to jointly complete user-specified computational operations.

[0041] As can be seen, in this embodiment, the encoding end first divides the video frame image to be encoded into at least two block images. Then, it identifies the region of interest (ROI) in each block image. Based on the target scaling ratio of the video frame image and the attribute information of the ROI in the block image, it determines the sub-scaling ratio of that block image. Then, it encodes the block image based on the sub-scaling ratio to obtain the encoded image of the video frame image. Thus, the sub-scaling ratio in each block image can be different, and the sub-scaling ratio is related to the attribute information of the ROI in the block image, which can avoid the loss of image details in the ROI and improve the image display effect. On the other hand, the decoding end can decode the received encoded image based on the mode information of the received block image to obtain the restored video frame image, further improving the image display effect. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 A flowchart illustrating a video encoding method provided in an embodiment of this application;

[0044] Figure 2 A flowchart illustrating a video decoding method provided in an embodiment of this application;

[0045] Figure 3 This is a schematic diagram of the structure of a video encoding device provided in an embodiment of this application;

[0046] Figure 4 This is a schematic diagram of the structure of a video decoding device provided in an embodiment of this application;

[0047] Figure 5 A schematic diagram of the structure of a computer device provided in an embodiment of this application;

[0048] Figure 6 This is a schematic diagram of the structure of a combined processing device provided in an embodiment of this application;

[0049] Figure 7 This is a schematic diagram of another combined processing device provided in an embodiment of this application. Detailed Implementation

[0050] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0051] The terms "first," "second," "third," and "fourth," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0052] In this document, the term "embodiment" means that a particular feature, result, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0053] The computer devices mentioned in this application may include servers, smartphones (such as Android phones, iOS phones, Windows Phones, etc.), tablet computers, handheld computers, desktop computers, laptop computers, mobile internet devices (MIDs) or wearable devices, etc. The above computer devices are merely examples and not an exhaustive list.

[0054] Optionally, the computer device includes an artificial intelligence processing unit. The processor of the artificial intelligence processing unit may include a general-purpose processor and an artificial intelligence processor.

[0055] The general-purpose processor may include one or more of the following: a central processing unit (CPU), a graphics processing unit (GPU), and / or an image processing unit (IPU).

[0056] Artificial intelligence processors include machine learning processing units (MLUs), and multiple MLUs can be integrated to form a multi-core artificial intelligence processor.

[0057] It should be noted that the computer equipment can be an encoding device, such as a video encoding device. Alternatively, the computer equipment can be a decoding device, such as a video decoding device; this is not a limitation.

[0058] The specific process of the method provided in the embodiments of this application will be further described below.

[0059] Please see Figure 1 , Figure 1 This is a flowchart illustrating a video encoding method provided in an embodiment of this application. The method is applied at the encoding end. Specifically, the method includes the steps shown in S101 to S105, wherein:

[0060] S101: The encoding end divides the video frame image to be encoded into at least two block images.

[0061] This application does not limit the video frame image; it can be any frame image in the video data to be encoded. For example, the first frame image in the video data, or images after the first frame image, etc.

[0062] In this embodiment, the block image can be a portion of a video frame image. The number of block images can be greater than or equal to 2, and this application does not limit the number of block images. Furthermore, this application does not limit the size of the block images; optionally, the block image is a sub-block corresponding to 64*64 pixels.

[0063] In one possible example, step S101 may include the following steps: the encoding end determines the size of the block image based on the resolution of the video frame image to be encoded; the encoding end divides the video frame image based on the size to obtain at least two block images.

[0064] The resolution of a video frame determines the level of detail in the image it displays. Generally, higher resolution means more pixels, resulting in a clearer image and better print quality. However, it also increases file storage space. Therefore, to improve video transmission speed, lower resolution is often used for transmission.

[0065] The size of the block image can be described in pixels or in distance units, etc., and there is no limitation here.

[0066] In this example, the encoding end first determines the size of the block image based on the resolution of the video frame image to be encoded. Then, based on this size, the video frame image is divided into at least two block images. This improves the effectiveness of block image division, which in turn enhances the display quality of the encoded image.

[0067] It should be noted that the areas of the block images can be equal or unequal. The block images obtained by dividing the image based on its size should be smaller than or equal to that size.

[0068] S102: The encoding end identifies the region of interest in the block image.

[0069] In this embodiment, the Region of Interest (ROI) refers to the image region in the block image corresponding to the object to be identified. In other words, the ROI is the region to be processed. For example, if the object to be identified is a face, then the ROI is the region in the block image corresponding to the face image, etc. This application does not limit the object to be identified, the shape of the ROI, or the method for identifying the ROI. The shape of the ROI can be a shape outlined by a rectangle, circle, ellipse, irregular polygon, etc.

[0070] Optionally, step S102 may include: the encoding end using a preset neural network to identify the region of interest in the block image.

[0071] The pre-defined neural network may include an input layer for receiving input block images. The pre-defined neural network may also include at least two shared convolutional layers connected to the input layer to obtain the correlation between attributes between the shared convolutional layers, which helps improve recognition accuracy. For example, men generally have short hair, and women generally have long hair; that is, there is a correlation between gender and hair length. Shared convolutional layers allow the machine to learn the relationship between these two attributes.

[0072] Each shared convolutional layer comprises an input layer, intermediate layers, and an output layer. The input layer of each layer is connected to the output layer of the previous layer, and the number of output neurons in the final output layer matches the expected number of output neurons. The intermediate layers can include convolution operations, normalization operations, activation operations, max pooling, or average pooling operations. At least two shared convolutional layers can yield at least two global feature maps and corresponding basic attributes for the block image. These basic attributes refer to shallow features in the video frame image; for example, if the type to be identified is a face, the basic attributes could be the face's contours and facial features.

[0073] The pre-defined neural network may further include a region pooling layer connected to a shared convolutional layer and an output layer connected to the region pooling layer. The region pooling layer is used to segment the global feature map to obtain at least two local feature maps. Each local feature map includes at least one region of interest. The output layer is used to output the region of interest in the block image.

[0074] It is understandable that using a pre-set neural network to identify regions of interest in block images can help improve the accuracy of identifying regions of interest.

[0075] S103: The encoding end determines the sub-scaling ratio of the block image based on the target scaling ratio of the video frame image and the attribute information of the region of interest.

[0076] In this embodiment, the target scaling ratio refers to the scaling ratio of the resolution in the video frame image, which can be equal to the ratio of the resolution between the encoded video frame image and the unencoded video frame image. This application does not limit the size of the target scaling ratio. Optionally, before step S103, the following steps are included: the encoding end determines the network transmission status of the encoding end; and the target scaling ratio of the video frame image is determined based on the network transmission status and the reference scaling ratio of the reference frame image corresponding to the video frame image.

[0077] The network transmission status includes the current network transmission rate, the upper and lower limits of the network transmission rate, and the remaining duration of the currently transmitted file. It can be understood that the transmission of encoded images depends on the network transmission status used by the encoding end. In this example, determining the target scaling ratio based on the network transmission rate can improve the accuracy of video frame image compression and the image display effect.

[0078] A reference frame image refers to the reference image of that video frame image in the corresponding video file. In other words, encoding is performed based on the reference frame image. This reference frame image can be the previous frame, several previous frames, or several subsequent frames of the video frame image in the video file, etc., without limitation. It should be noted that the reference frame image is an encoded image, and it contains images similar to those in the video frame image.

[0079] The reference scaling ratio refers to the scaling ratio of the reference frame image. This application does not limit the method for determining the target scaling ratio based on the reference scaling ratio of the reference frame image and the network transmission status. If the encoder's network transmission status supports image transmission at the reference scaling ratio, the target scaling ratio can be the reference scaling ratio or a value smaller than the reference scaling ratio, thereby improving the image display effect. If the encoder's network transmission status does not support image transmission at the reference scaling ratio, the target scaling ratio can be a value larger than the reference scaling ratio, thereby increasing the image transmission rate.

[0080] It is understandable that, in this example, determining the target scaling ratio of the video frame based on the reference scaling ratio of the reference frame image and the network transmission status of the encoder is beneficial to improving the image display effect and transmission rate.

[0081] In this embodiment, the attribute information of the region of interest may include the area size, object type, relative position, etc., which are not limited here. It can be understood that the larger the area of ​​the region of interest, the more details it carries, and the less the image of the region of interest can be scaled, so as to retain more details.

[0082] Object type refers to the type of object in the region of interest. For example, if the object to be identified is a human face, the object type could be eyes, nose, mouth, eyebrows, facial contours, etc. It can be understood that the greater the probability of object type variation, the greater the difference between the representation and the reference frame image. Therefore, the region of interest can be stretched as little as possible to retain more detail. For example, among eyes, nose, and mouth, the nose has the lowest probability of variation, while the mouth has the highest. Therefore, the sub-scaling ratio for the eyes can be greater than that for the nose, and smaller than that for the mouth.

[0083] Relative position refers to the position of an object within a region of interest relative to the position of the block image, or the position of an object within a region of interest relative to the entire object. It can be understood that the more centered an image is within a block image, the more detail it contains. Therefore, the sub-scaling ratio of the block image can be set based on the relative position of the region of interest to improve the image display effect.

[0084] In this embodiment, each image block corresponds to a sub-scaling ratio, which can be greater than or less than the target scaling ratio. It should be noted that the weighted average of the sub-scaling ratios equals the target scaling ratio. This allows the image to be displayed at the overall scaling size of the target scaling ratio, improving the image display effect.

[0085] In one possible example, if the areas of all image blocks are equal, then the preset weights of the sub-scaling ratios of each image block are equal, and the average value among the sub-scaling ratios of each image block equals the target scaling ratio. If the areas of the image blocks are unequal, the preset weights of the sub-scaling ratios of each image block can be determined by the attribute information of that image block. The attribute information of the image block may include its size, object type, and relative position, etc., which can be referred to in the description of the attribute information of the region of interest, and will not be elaborated here. It is understood that determining the preset weights of the image blocks based on their attribute information helps improve the accuracy of the preset weight settings and facilitates better image display.

[0086] This application does not limit the method for determining the sub-scaling ratio of the block image. In one possible example, step S103 may include the following steps: the encoding end calculates the interest value of the block image based on the attribute information of the region of interest; the encoding end determines the sub-scaling ratio of the block image based on the interest value of the block image and the target scaling ratio of the video frame image.

[0087] In this embodiment, the interest value of a block image is used to describe the degree of interest in a region of interest within the block image. It can be calculated based on the attribute information of the region of interest; for example, the larger the area of ​​the region of interest, the higher the interest value; the greater the probability of change in the object type within the region of interest, the higher the interest value; the more centrally located the region of interest, the higher the interest value, etc.

[0088] In this example, the encoder first calculates the interest value of a block image based on the attribute information of the region of interest (ROI). Then, based on the magnitude of this interest value and the target scaling ratio of the video frame image, it determines the sub-scaling ratio of each block image within the video frame. Thus, the sub-scaling ratio in each block image can be different, and since the sub-scaling ratio is related to the attribute information of the ROI in the block image, it avoids the loss of image details in the ROI and improves the image display effect.

[0089] In this embodiment, the target scaling ratio can be allocated based on the principle that the larger the interest value of the block image, the smaller the sub-scaling ratio of the block image, and the weighted average of the sub-scaling ratios equals the target scaling ratio, so as to obtain the sub-scaling ratio of each block image. No limitation is made here.

[0090] S104: The encoding end encodes the block image based on the sub-scaling ratio of the block image to obtain the encoded image of the video frame image.

[0091] This application does not limit the encoding method for block images. It can be based on variable-length coding algorithms or arithmetic coding algorithms, or it can be based on context-adaptive variable-length coding algorithms or context-adaptive binary arithmetic coding algorithms, etc. It can be understood that encoding block images based on the sub-scaling ratio of the block image can obtain an encoded image for each block image. The sub-scaling ratio of each encoded image can be different and related to the attribute information of the region of interest in the block image. This can avoid the loss of image details in the region of interest and improve the image display effect.

[0092] In one possible example, step S104 may include the following steps: the encoder acquires the binary data corresponding to the block image; the encoder models based on the context information of the block image to obtain a context model; the encoder performs binary arithmetic encoding on the binary data based on the sub-scaling ratio of the block image and the context model to obtain the encoded image of the video frame image.

[0093] The binary data corresponding to the block image may include the binary data corresponding to each pixel in the block image, or it may include the binary data corresponding to the coding unit of the block image, etc., without limitation.

[0094] The context information of a block image refers to the image information of the reference block image corresponding to the block image in the reference frame image corresponding to the block image. This can include the position information, motion vectors, and residual data of the reference block image, etc., without limitation. The context model is a model derived from the context information of the block image and is used to encode the block image. It can be understood that encoding the block image based on the context model can improve the efficiency and accuracy of the encoding.

[0095] In this example, the encoder first obtains the binary data corresponding to the block image and the context model corresponding to the context information of the block image. Then, based on the sub-scaling ratio of the block image and the context model, binary arithmetic encoding is performed on the binary data, which can improve the encoding efficiency and accuracy. Moreover, the scaling ratio of the block image in the encoded image is related to the attribute information of the region of interest in the block image, which can avoid the loss of image details in the region of interest and help improve the display effect of the image.

[0096] The encoding process involves first dividing the video frame image to be encoded into at least two block images. Then, the region of interest (ROI) is identified within each block image. Based on the target scaling ratio of the video frame image and the attribute information of the ROI in the block image, a sub-scaling ratio is determined for that block image. Finally, the block image is encoded based on this sub-scaling ratio to obtain the encoded video frame image. In this way, the sub-scaling ratio in each block image can be different, and the sub-scaling ratio is related to the attribute information of the ROI in the block image. This avoids the loss of image details in the ROI and improves the image display quality.

[0097] S105: The encoding end sends the mode information of the encoded image and the block image to the decoding end.

[0098] In this embodiment, the mode information of the block image includes at least the sub-scaling ratio of the block image. Thus, the decoding end can reconstruct the encoded sub-images corresponding to the block images in the encoded image based on the sub-scaling ratio to obtain the reconstructed video frame image, further improving the image display effect.

[0099] The mode information of the block image may also include information about the reference frame image corresponding to the video frame image, such as indicating that the reference frame image is the previous frame image of the video frame image. If the encoding is based on affine motion compensation technology, the mode information may also include the size of the affine transformation unit, the position and direction of the motion vector, and the precision, etc., which are not limited here. It can be understood that the decoding end decodes the received encoded image based on the mode information of the received block image to obtain the reconstructed block image, thereby obtaining the reconstructed video frame image, which can further improve the image display effect.

[0100] Please see Figure 2 , Figure 2 This is a flowchart illustrating a video decoding method provided in an embodiment of this application. The method is applied at the decoding end. Specifically, the method includes the steps shown in steps S201 and S202, wherein:

[0101] S201: The decoding end receives the encoded image of the video frame image and the mode information of each block image in the video frame image from the encoding end.

[0102] In this embodiment, the mode information of the block image includes at least the sub-scaling ratio of the block image. This mode information may also include information about a reference frame image corresponding to the video frame image, such as indicating that the reference frame image is the previous frame image of the video frame image. If the encoding is based on affine motion compensation technology, the mode information may also include the size of the affine transformation unit, the position and direction of the motion vector, and the accuracy, etc., which are not limited here.

[0103] S202: The decoding end decodes the encoded image based on the mode information to obtain the video frame image.

[0104] It is understandable that after receiving the encoded image and block image mode information of the video frame, the decoding end can decode the encoded image based on the mode information to obtain the reconstructed video frame image. Displaying the decoded image in this way can improve the image display quality.

[0105] The methods of the embodiments of this application have been described in detail above, and the apparatus of the embodiments of this application is provided below.

[0106] Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of a video encoding device 300 according to an embodiment of this application. Figure 3 As shown, the video encoding device 300 includes:

[0107] The partitioning unit 301 is used to partition the video frame image to be encoded to obtain at least two block images;

[0108] The recognition unit 302 is used to identify the region of interest in the block image;

[0109] The determining unit 303 is used to determine the sub-scaling ratio of the block image based on the target scaling ratio of the video frame image and the attribute information of the region of interest;

[0110] The encoding unit 304 is used to encode the block image based on the sub-scaling ratio of the block image to obtain the encoded image of the video frame image;

[0111] The sending unit 305 sends the mode information of the encoded image and the block image to the decoding end, wherein the mode information of the block image includes at least the sub-scaling ratio of the block image.

[0112] In one possible example, the determining unit 303 is specifically used to calculate the interest value of the block image based on the attribute information of the region of interest, wherein the attribute information includes at least one of the following: area size, object type, and relative position; and to determine the sub-scaling ratio of the block image based on the interest value of the block image and the target scaling ratio of the video frame image, wherein the larger the interest value of the block image, the smaller the sub-scaling ratio of the block image, and the weighted average of the sub-scaling ratios is equal to the target scaling ratio.

[0113] In one possible example, if the areas of the block images are equal, then the preset weights of the sub-scaling ratios are equal; if the areas of the block images are unequal, then the preset weights of the sub-scaling ratios are determined by the attribute information of the block images.

[0114] In one possible example, the partitioning unit 301 is specifically used to determine the size of the block image based on the resolution of the video frame image to be encoded; and to partition the video frame image based on the size to obtain at least two block images.

[0115] In one possible example, the encoding unit 304 is specifically used to acquire the binary data corresponding to the block image; to model based on the context information of the block image to obtain a context model; and to perform binary arithmetic encoding on the binary data based on the sub-scaling ratio of the block image and the context model to obtain the encoded image of the video frame image.

[0116] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of a video decoding device 400 according to an embodiment of this application. Figure 4 As shown, the video decoding device 400 includes:

[0117] The receiving unit 401 is used to receive the encoded image of the video frame image and the mode information of each block image in the video frame image from the encoding end. The mode information of the block image includes at least the sub-scaling ratio of the block image.

[0118] The decoding unit 402 is used to decode the encoded image based on the mode information to obtain the video frame image.

[0119] It should be noted that the implementation of each module can also be referenced accordingly. Figure 1 or Figure 2 The corresponding description of the method embodiments shown.

[0120] and Figure 1 or Figure 2 The embodiments shown are consistent; please refer to [link / reference]. Figure 5 , Figure 5 This is a schematic diagram of the structure of a computer device proposed in this application, such as... Figure 5 As shown, the computer device includes a processor, memory, and a communication interface. The processor may include a general-purpose processor and an artificial intelligence processor. The processor 510, memory 520, and communication interface 530 are interconnected via a bus 550.

[0121] The aforementioned memory 520 stores a computer program 540 and is configured to be executed by the aforementioned processor 510. If the computer device is an encoding terminal, then... Figure 3 The functions implemented by the transmitting unit 305 shown can be realized through the communication interface 530. Figure 3 The functions implemented by the segmentation unit 301, identification unit 302, determination unit 303, and encoding unit 304 shown can be implemented by the processor 510. The computer program 540 includes instructions for performing the following steps:

[0122] The video frame image to be encoded is divided into at least two block images;

[0123] Identify the region of interest in the block image;

[0124] The sub-scaling ratio of the block image is determined based on the target scaling ratio of the video frame image and the attribute information of the region of interest;

[0125] The block image is encoded based on the sub-scaling ratio of the block image to obtain the encoded image of the video frame image;

[0126] The mode information of the encoded image and the block image is sent to the decoding end, wherein the mode information of the block image includes at least the sub-scaling ratio of the block image.

[0127] In one possible example, in determining the sub-scaling ratio of the block image based on the target scaling ratio of the video frame image and the attribute information of the region of interest, the computer program 540 includes instructions for performing the following steps:

[0128] The interest value of the block image is calculated based on the attribute information of the region of interest, wherein the attribute information includes at least one of the following: area size, object type, and relative position;

[0129] The sub-scaling ratio of the block image is determined based on the interest value of the block image and the target scaling ratio of the video frame image, wherein the larger the interest value of the block image, the smaller the sub-scaling ratio of the block image, and the weighted average of the sub-scaling ratios is equal to the target scaling ratio.

[0130] In one possible example, if the areas of the block images are equal, then the preset weights of the sub-scaling ratios are equal; if the areas of the block images are unequal, then the preset weights of the sub-scaling ratios are determined by the attribute information of the block images.

[0131] In one possible example, regarding the division of the video frame image to be encoded into at least two block images, the computer program 540 includes instructions for performing the following steps:

[0132] The size of the block image is determined based on the resolution of the video frame image to be encoded;

[0133] The video frame image is divided based on the size to obtain at least two block images.

[0134] In one possible example, regarding the encoding of the block image based on a sub-scaling ratio of the block image to obtain the encoded image of the video frame image, the computer program 540 includes instructions for performing the following steps:

[0135] Obtain the binary data corresponding to the block image;

[0136] A context model is obtained by modeling based on the context information of the block image;

[0137] Based on the sub-scaling ratio of the block image and the context model, binary arithmetic encoding is performed on the binary data to obtain the encoded image of the video frame image.

[0138] If the computer device is the decoding end, then Figure 4 The functions implemented by the receiving unit 401 shown can be realized through the communication interface 530. Figure 4 The functions implemented by the decoding unit 402 shown can be implemented by the processor 510. The computer program 540 includes instructions for performing the following steps:

[0139] The encoded image of the video frame image and the mode information of each block image in the video frame image are received from the encoding end. The mode information of the block image includes at least the sub-scaling ratio of the block image.

[0140] The encoded image is decoded based on the pattern information to obtain the video frame image.

[0141] This application also provides a computer-readable storage medium that stores a computer program, which is executed by a processor to implement some or all of the steps of any of the video encoding or video decoding methods described in the above method embodiments.

[0142] This application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps of any of the video encoding or video decoding methods described in the above method embodiments.

[0143] Please refer to Figure 6 , Figure 6 This is a schematic diagram of a combined processing device provided in an embodiment of this application. Figure 6 As shown, the combined processing unit includes a video encoding device, a universal interconnect interface, and other processing devices. The video encoding device interacts with the other processing devices to jointly complete the user-specified operations.

[0144] Other processing devices include one or more processor types such as CPUs, GPUs, and neural network processors, as well as general-purpose / dedicated processors. There is no limit to the number of processors included in other processing devices. These other processing devices serve as interfaces between the video encoding device and external data and control, including data transfer and basic control functions such as starting and stopping the video encoding device. Other processing devices can also collaborate with the video encoding device to complete computational tasks.

[0145] A general interconnect interface is used to transmit data and control commands between the video encoding device and other processing devices. The video encoding device obtains the required input data from other processing devices and writes it to the on-chip storage device; it can also obtain control commands from other processing devices and write them to the on-chip control buffer; and it can read data from the storage module of the video encoding device and transmit it to other processing devices.

[0146] In one possible example, such as Figure 6 As shown, the combined processing apparatus may further include a storage device, which is connected to both the video encoding apparatus and the other processing apparatus. The storage device is used to store data stored in the video encoding apparatus and the other processing apparatus, and is particularly suitable for data that cannot be fully stored in the internal storage of the video encoding apparatus or other processing apparatus, requiring computation.

[0147] Please refer to Figure 7 , Figure 7 This is a schematic diagram of another combined processing device provided in an embodiment of this application. Figure 7 As shown, the combined processing unit includes a video decoding unit, a general interconnect interface, other processing units, and a storage unit. The video decoding unit interacts with the other processing units to jointly complete the user-specified operations.

[0148] Among them, the general interconnect interface, other processing devices, and storage devices can be referred to Figure 6 The description states that other processing devices serve as interfaces between the video decoding device and external data and control, including data transfer and basic control of the video encoding device, such as starting and stopping it; other processing devices can also cooperate with the video decoding device to complete computational tasks.

[0149] The universal interconnect interface is used to transmit data and control commands between the video decoding device and other processing devices. The video decoding device obtains the required input data from other processing devices and writes it to the on-chip storage; it can also obtain control commands from other processing devices and write them to the on-chip control buffer; and it can read data from the video decoding device's storage module and transmit it to other processing devices.

[0150] The storage device is connected to both the video decoding device and the other processing devices. The storage device is used to store data stored in the video decoding device and the other processing devices, and is particularly suitable for data that cannot be fully stored in the internal storage of the video decoding device or other processing devices, requiring computation.

[0151] In this embodiment, the combined processing device can serve as a system-on-a-chip (SoC) for devices such as mobile phones, robots, drones, and video surveillance equipment, effectively reducing the core area of ​​the control section, increasing processing speed, and reducing overall power consumption. In this case, the general interconnect interface of the combined processing device is connected to certain components of the device. These components include, for example, cameras, displays, mice, keyboards, network cards, and Wireless-Fidelity (Wi-Fi) interfaces.

[0152] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0153] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0154] In the several embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical or other forms.

[0155] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0156] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software program module.

[0157] If the integrated unit is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0158] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage device, which may include: a flash drive, a read-only memory, a random access memory, a magnetic disk, or an optical disk, etc.

[0159] The embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A video encoding method, characterized in that, include: The video frame image to be encoded is divided into at least two block images; Identify the region of interest in the block image; The sub-scaling ratio of the block image is determined based on the target scaling ratio of the video frame image and the attribute information of the region of interest; The block image is encoded based on the sub-scaling ratio of the block image to obtain the encoded image of the video frame image; The mode information of the encoded image and the block image is sent to the decoding end, wherein the mode information of the block image includes at least the sub-scaling ratio of the block image; The step of determining the sub-scaling ratio of the block image based on the target scaling ratio of the video frame image and the attribute information of the region of interest includes: The interest value of the block image is calculated based on the attribute information of the region of interest, wherein the attribute information includes at least one of the following: area size, object type, and relative position. The relative position refers to the position of the object in the region of interest relative to the position of the block image, or the relative position refers to the position of the object in the region of interest relative to the position of the entire object. The interest value of the block image is used to describe the degree of interest of the region of interest in the block image. The interest value of the block image is positively correlated with the area size and positively correlated with the probability of change of the object type. The more central the relative position, the greater the interest value of the block image. The sub-scaling ratio of the block image is determined based on the interest value of the block image and the target scaling ratio of the video frame image. The larger the interest value of the block image, the smaller the sub-scaling ratio of the block image. The weighted average of the sub-scaling ratios is equal to the target scaling ratio. The larger the area, the smaller the sub-scaling ratio; the greater the probability of change in the object type, the smaller the sub-scaling ratio; and the more centered the relative position, the smaller the sub-scaling ratio.

2. The method according to claim 1, characterized in that, in, If the areas of the block images are equal, then the preset weights of the sub-scaling ratios are equal; if the areas of the block images are unequal, then the preset weights of the sub-scaling ratios are determined by the attribute information of the block images.

3. The method according to any one of claims 1 or 2, characterized in that, The video frame image to be encoded is divided into at least two block images, including: The size of the block image is determined based on the resolution of the video frame image to be encoded; The video frame image is divided based on the size to obtain at least two block images.

4. The method according to any one of claims 1 or 2, characterized in that, The process of encoding the block image based on the sub-scaling ratio of the block image to obtain the encoded image of the video frame image includes: Obtain the binary data corresponding to the block image; A context model is obtained by modeling based on the context information of the block image; Based on the sub-scaling ratio of the block image and the context model, binary arithmetic encoding is performed on the binary data to obtain the encoded image of the video frame image.

5. A video decoding method, characterized in that, include: The encoded image of the video frame image and the mode information of each block image in the video frame image are received from the encoding end. The mode information of the block image includes at least the sub-scaling ratio of the block image. The encoded image is decoded based on the pattern information to obtain the video frame image; The sub-scaling ratio of the block image is determined based on the target scaling ratio of the video frame image and the attribute information of the region of interest, specifically including: The interest value of the block image is calculated based on the attribute information of the region of interest, wherein the attribute information includes at least one of the following: area size, object type, and relative position. The relative position refers to the position of the object in the region of interest relative to the position of the block image, or the relative position refers to the position of the object in the region of interest relative to the position of the entire object. The interest value of the block image is used to describe the degree of interest of the region of interest in the block image. The interest value of the block image is positively correlated with the area size and positively correlated with the probability of change of the object type. The more central the relative position, the greater the interest value of the block image. The sub-scaling ratio of the block image is determined based on the interest value of the block image and the target scaling ratio of the video frame image, wherein the larger the interest value of the block image, the smaller the sub-scaling ratio of the block image, and the weighted average of the sub-scaling ratios is equal to the target scaling ratio. The larger the area, the smaller the sub-scaling ratio; the greater the probability of change in the object type, the smaller the sub-scaling ratio; and the more centered the relative position, the smaller the sub-scaling ratio.

6. A video encoding device, characterized in that, Includes a unit for performing the video encoding method as described in any one of claims 1-4.

7. A video decoding device, characterized in that, It includes a unit for performing the video decoding method as described in claim 5.

8. A computer device, comprising a processor, a memory, and a communication interface, wherein, The memory stores a computer program configured to be executed by the processor, the computer program including methods for performing any one of claims 1-5.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that causes a computer device to perform the method according to any one of claims 1-5.

Images