Multimedia coding method and apparatus, and device, medium and product
By recording and utilizing the transformation mode information of residual blocks in multimedia encoding, the encoding mode is directly determined, which solves the problem of high computational complexity of transformation mode information selection in the prior art, and improves the encoding efficiency.
Patent Information
- Application Number
- PCT/CN2024/090763
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-04
- Filing Date
- 2024-04-30
- Publication Date
- 2024-11-07
AI Technical Summary
The existing multimedia encoding technology has high computational complexity when selecting transform mode information, resulting in increased encoding time and resource overhead, affecting efficiency.
During the multiple encoding process, the transformation mode information of each residual block is recorded in the first encoding stage, and the transformation mode information is determined in the non-first encoding stage based on the recorded information and the attributes of the currently coded residual block, and directly encoded to avoid Both modes are encoded simultaneously, thereby reducing the computational complexity.
It effectively reduces the overhead of code stream encoding time and computing resources, and improves coding efficiency.
Smart Images

Figure CN2024090763_07112024_PF_FP_ABST
Abstract
Description
Multimedia coding method, device, equipment, medium and product
[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on May 4, 2023, with application number 2023105032807 and application name “Multimedia Coding Methods, Devices, Equipment, Media and Products”. The entire contents of the application are incorporated by reference into this application. Technical Field
[0002] The present application relates to the field of multimedia coding technology, and more specifically, to a multimedia coding method, a multimedia coding device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art
[0003] Currently, there are many encoders that use standardized video coding technologies. The most common encoders must go through the main steps of prediction, transformation, quantization, and entropy coding in sequence to obtain a compressed binary code stream. In some scenarios, for transform units that meet certain size restrictions, the transformation step is optional.
[0004] In related art, an encoder determines whether to skip transforms by encoding in both transform mode and transform skip mode simultaneously, then selecting the most appropriate mode based on the encoding results. However, encoding in both modes simultaneously increases the encoder's computational complexity, requiring more computing resources and increasing encoding time.
[0005] Summary of the Invention
[0006] To solve the above-mentioned technical problems, the embodiments of the present application provide a multimedia encoding method, a multimedia encoding device, an electronic device, a computer-readable storage medium and a computer program product, which can effectively reduce the computational complexity of selecting transformation mode information in multiple encodings, thereby reducing the bitstream encoding time and computing resource overhead, and helping to improve encoding efficiency.
[0007] According to one aspect of an embodiment of the present application, a multimedia encoding method is provided, comprising: in a first encoding stage in multiple encodings of multimedia, recording transform mode information of each residual block, the transform mode information indicating whether a transform skip mode is applied; in a non-first encoding stage, obtaining the transform mode information of each residual block recorded in the first encoding stage; determining the transform mode information of the residual block to be encoded currently based on the transform mode information recorded in the first encoding stage and attribute information of the residual block to be encoded currently; and encoding the residual block to be encoded currently based on the transform mode information of the residual block to be encoded currently.
[0008] According to one aspect of an embodiment of the present application, a multimedia encoding device is provided, including: a recording module for recording transform mode information of each residual block in the first encoding stage of multiple encodings of multimedia, the transform mode information indicating whether a transform skip mode is applied; an acquisition module for acquiring the transform mode information of each residual block recorded in the first encoding stage in a non-first encoding stage; a determination module for determining the transform mode information of the residual block to be encoded currently based on the transform mode information recorded in the first encoding stage and the attribute information of the residual block to be encoded currently; and an encoding module for encoding the residual block to be encoded currently based on the transform mode information of the residual block to be encoded currently.
[0009] According to one aspect of an embodiment of the present application, an electronic device is provided, comprising one or more processors; a storage device for storing one or more computer programs, which, when executed by one or more processors, enables the electronic device to implement the multimedia encoding method as described above.
[0010] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor of an electronic device, the electronic device executes the multimedia encoding method as described above.
[0011] According to one aspect of an embodiment of the present application, a computer program product is provided, including a computer program, wherein the computer program is stored in a computer-readable storage medium, and a processor of an electronic device reads and executes the computer program from the computer-readable storage medium, so that the electronic device performs the multimedia encoding method as described above. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] The accompanying drawings are incorporated into and constitute a part of the specification, illustrating embodiments consistent with the present application and, together with the specification, serving to explain the principles of the present application. Obviously, the drawings described below are merely some embodiments of the present application, and a person of ordinary skill in the art can derive other drawings based on these drawings without inventive effort. In the drawings:
[0013] FIG1 is a schematic diagram of an implementation environment involved in this application;
[0014] FIG2 is a flowchart of a multimedia encoding method shown in an exemplary embodiment of the present application;
[0015] FIG3 is a schematic diagram showing a storage of transform mode information of a residual block according to an exemplary embodiment of the present application;
[0016] FIG4 is a flow chart of another multimedia encoding method shown in an exemplary embodiment of the present application;
[0017] FIG5 is a flowchart of another multimedia encoding method shown in an exemplary embodiment of the present application;
[0018] FIG6 is a schematic diagram showing a distribution of change patterns of sub-blocks of a residual block according to an exemplary embodiment of the present application;
[0019] FIG7 is a flowchart of another multimedia encoding method shown in an exemplary embodiment of the present application;
[0020] FIG8 is a flowchart of another multimedia encoding method shown in an exemplary embodiment of the present application;
[0021] FIG9 is a flowchart of a multimedia encoding method shown in another exemplary embodiment of the present application;
[0022] FIG10 is a flowchart of another multimedia encoding method shown in another exemplary embodiment of the present application;
[0023] FIG11 is a flowchart of another multimedia encoding method shown in an exemplary embodiment of the present application;
[0024] FIG12A is a schematic diagram of an area of a predetermined position of a residual block shown in an exemplary embodiment of the present application;
[0025] FIG12B is a schematic diagram of another predetermined position area of a residual block according to an exemplary embodiment of the present application;
[0026] FIG13 is a flowchart of another multimedia encoding method shown in an exemplary embodiment of the present application;
[0027] FIG14 is a flowchart of a multimedia encoding method shown in another exemplary embodiment of the present application;
[0028] FIG15 is a flowchart of another multimedia encoding method shown in another exemplary embodiment of the present application;
[0029] FIG16 is a schematic diagram showing a flag bit corresponding to the first encoding of the current residual to be encoded, shown in another exemplary embodiment of the present application;
[0030] FIG17 is a flowchart of a multimedia encoding method shown in another exemplary embodiment of the present application;
[0031] FIG18 is a structural block diagram of a multimedia encoding device shown in another exemplary embodiment of the present application;
[0032] FIG19 shows a schematic structural diagram of a computer system suitable for implementing an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION
[0033] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, identical numerals in different figures represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.
[0034] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.
[0035] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.
[0036] It should also be noted that the term "plurality" used in this application refers to two or more. "And / or" describes the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.
[0037] The technical solutions of the embodiments of the present application relate to the field of cloud technology. Before introducing the technical solutions of the embodiments of the present application, a brief introduction to cloud technology is first given.
[0038] Cloud technology is a general term for network technologies, information technologies, integration technologies, management platform technologies, and application technologies based on the cloud computing business model. It can form a resource pool that can be used flexibly and conveniently on demand. Cloud computing technology will become a crucial support. Backend services for technical network systems, such as video websites, image websites, and more portals, require extensive computing and storage resources. With the rapid development and application of the internet industry, every item will likely have its own unique identifier, requiring transmission to backend systems for logical processing. Different levels of data will be processed separately, and data from various industries will require a strong system backend, which can only be achieved through cloud computing.
[0039] Cloud computing is a computing model that distributes computing tasks across a resource pool consisting of a large number of computers, enabling various application systems to access computing power, storage space, and information services as needed. The network that provides these resources is called the "cloud." To users, these resources appear infinitely scalable and can be accessed at any time, used on demand, expanded at any time, and paid for on a per-use basis.
[0040] As a provider of cloud computing infrastructure, a cloud computing resource pool (referred to as a cloud platform, generally referred to as an IaaS (Infrastructure as a Service) platform) is established. Various types of virtual resources are deployed in the resource pool for external customers to choose and use. The cloud computing resource pool mainly includes: computing devices (virtualized machines, including operating systems), storage devices, and network devices.
[0041] The following is a detailed description of the technical solutions of the embodiments of this application in combination with cloud technology and implementation environment:
[0042] Please refer to Figure 1, which is a schematic diagram of an implementation environment involved in this application. It includes a content producer 10 and a server 20. In one example, server 20 can be a computing device in the aforementioned cloud computing resource pool, which has a large amount of computing and storage resources and includes an encoder. Content producer 10 can be a video website, an image website, or the like.
[0043] The content producer 10 generates multimedia and sends a request to the server 20 , which carries the multimedia. In one example, the content producer 10 also needs to pay to request the server 20 to encode the multimedia multiple times using its own resources.
[0044] The server 20 is used to encode the multimedia multiple times, wherein, during the first encoding, steps such as prediction, transformation, quantization, and entropy coding are performed, and then the transformation mode information of the residual block obtained after the prediction is recorded. Specifically, in the first encoding stage of the multimedia multiple encoding process, the transformation mode information of the residual block is recorded, and in the non-first encoding stage, the transformation mode information of each residual block recorded in the first encoding stage is obtained; based on the transformation mode information recorded in the first encoding stage and the attribute information of the residual block to be encoded currently, the transformation mode information of the residual block to be encoded currently is determined; and the residual block to be encoded currently is encoded based on the transformation mode information of the residual block to be encoded currently.
[0045] In other embodiments, the multimedia encoding method can also be independently executed by the content producer 10. For example, the content producer is provided with an encoder. After generating multimedia through the media generator, the encoder is used to perform multiple encodings. In the first encoding stage of the multimedia multiple encoding process, the encoder of the content producer records the transformation mode information of the residual block, and in the non-first encoding stage, obtains the transformation mode information of each residual block recorded in the first encoding stage, and then determines the transformation mode information of the residual block to be encoded in combination with the attribute information of the residual block to be encoded, and then encodes the residual block to be encoded.
[0046] In other embodiments, the server 20 may also select multimedia by itself, for example, grab multimedia from the network, and then encode the multimedia multiple times.
[0047] In one example, the aforementioned content producer 10 can be any electronic device that can obtain the target video and the image to be processed, such as a smart phone, a tablet, a laptop, a computer, an intelligent voice interaction device, a smart home appliance, a vehicle-mounted terminal, an aircraft, etc. The server 20 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network) and big data and artificial intelligence platforms. This is not restricted here.
[0048] 1 , the content producer 10 and the server 20 have previously established a communication connection via a network 30, so that the content producer 10 and the server 20 can communicate with each other via the network 30. The network 30 can be a wired network or a wireless network, and this is not limited here.
[0049] It should be noted that in the specific implementation of this application, if the multimedia involves object-related information, when the embodiment of this application is applied to a specific product or technology, it is necessary to obtain the object's permission or consent, and the collection, use and processing of relevant data must comply with the relevant regulations and standards of the relevant countries and regions.
[0050] The following describes in detail the various implementation details of the technical solutions of the embodiments of the present application:
[0051] As shown in FIG2 , FIG2 is a flowchart of a multimedia encoding method according to an embodiment of the present application. The method can be applied to the implementation environment shown in FIG1 . The method can be executed in an electronic device, for example, by a content producer or a server, or by both the content producer and the server. In the embodiment of the present application, the method is described by taking the server as an example. The multimedia encoding method may include steps S210 to S240, which are described in detail as follows:
[0052] S210 . In a first encoding stage of multiple encodings of multimedia, record transform mode information of each residual block, where the transform mode information indicates whether a transform skip mode is applied.
[0053] In an embodiment of the present application, multimedia is encoded multiple times, i.e., at least twice; for example, at different bit rates. Multimedia includes, but is not limited to, any of graphics, animation, and video, which are not limited herein. Transform skip mode is a mode in which no transform is performed. In other words, transform mode information may indicate whether transform skip mode or transform-performing mode is applied, i.e., whether a transform operation is performed. A residual block may represent the difference between a corresponding coding block and a prediction block.
[0054] During the first encoding stage of multiple multimedia encoding processes, prediction, transformation, quantization, and entropy coding operations are performed. The prediction operation reduces the amount of encoded information to obtain a residual block; the transformation operation concentrates the energy of the residual signal in the frequency domain to improve the compression efficiency of subsequent entropy coding; the quantization operation eliminates lossy information; and the entropy coding is used to compress the quantity.
[0055] In an embodiment of the present application, during the first encoding process, the server will perform a transform operation on each residual block and record the transform mode information of each residual block. The transform mode information is used to indicate whether the transform skip mode is applied, that is, whether the residual block performs a transform operation; wherein, the server can obtain the transform mode information from the transform coefficients of each residual block. Specifically, the transform coefficients are numerical values calculated during the transformation process, which represent the expression of the residual block in the transform domain; different transform modes will produce different transform coefficients, so the transform mode information can be obtained by analyzing the transform coefficients. It is understandable that during the first encoding process, after the residual block is obtained through the prediction operation, some residual blocks may not be transformed when the transform operation is performed. For example, in JPEG compression, commonly used transform operations are discrete cosine transform (DCT) and discrete wavelet transform (DWT). For DCT transform, the server can determine whether DCT transform is used by analyzing the distribution of transform coefficients. For DWT transformation, since its transformation coefficients are sparse, the server can determine whether DWT transformation is adopted by detecting zero values in the transformation coefficients, and then the server can record which residual blocks are transformed and which residual blocks are skipped.
[0056] S220 . In a non-initial encoding stage, obtain transform mode information of each residual block recorded in the initial encoding stage.
[0057] In the embodiment of the present application, the non-first encoding may be any encoding after the first encoding, and the server will obtain the transformation mode information of each residual block recorded in the first encoding stage.
[0058] In one example, after obtaining the transform mode information for each residual block recorded during the initial encoding phase, the server integrates the obtained transform mode information for the residual blocks. For example, the transform mode information for each residual block is sorted and arranged based on its position to obtain the transform mode information for the entire residual block of the multimedia for subsequent analysis. As shown in Figure 3, the prediction operation obtains residual blocks 1-4, which are located in the upper left corner, upper right corner, lower left corner, and lower right corner, respectively. The respective transform mode information is correspondingly set in the upper left corner area 301, the upper right corner area 302, the lower left corner area 303, and the lower right corner area 304.
[0059] S230 : Determine the transform mode information of the residual block to be encoded currently according to the transform mode information recorded in the first encoding stage and the attribute information of the residual block to be encoded currently.
[0060] During the non-initial encoding phase, a prediction operation is performed to obtain the current residual block to be encoded. However, the block partitioning results may differ between the initial encoding and non-initial encoding prediction operations. Therefore, the current residual block to be encoded may differ from the residual block in the initial encoding, and thus the attribute information of the current residual block to be encoded may also differ from that in the initial encoding. The attribute information of the current residual block to be encoded refers to information describing the characteristics of the current residual block to be encoded, and is used to describe various attributes of an object, such as size and position. It is understood that when the current residual block to be encoded is obtained through the prediction operation, the server already has access to the attribute information of the current residual block to be encoded.
[0061] In an embodiment of the present application, the transformation mode information recorded in the first encoding stage will be used as an important basis for determining whether the current residual to be encoded needs to be transformed. The server will determine the transformation mode information of the current residual block to be encoded based on the transformation mode information recorded in the first encoding stage and the attribute information of the current residual to be encoded, that is, determine whether the current residual block to be encoded needs to be transformed.
[0062] S240 : Encode the current residual block to be encoded according to the transformation mode information of the current residual block to be encoded.
[0063] In an embodiment of the present application, after determining the transform mode information of the current residual block to be encoded, if the transform mode information of the current residual block to be encoded is to use a transform operation, a transform operation is performed on the current residual block to be encoded, such as using a discrete cosine transform (DCT). This will convert the residual block from the pixel domain to the frequency domain and concentrate most of the energy in the residual block on a small number of frequency domain coefficients, so that these coefficients can be compressed and transmitted more efficiently. If the transform mode information of the current residual block to be encoded is to skip the transform operation, a quantization operation is performed on the current residual block to be encoded, which will approximate and round the coefficients in the residual block, making it easier to compress.
[0064] In one embodiment, if the transform mode information of the residual block to be encoded currently indicates the application of a skip transform mode, a quantization operation is performed on the residual block to be encoded currently without performing a transform operation. If the transform mode information of the residual block to be encoded currently indicates the application of a perform transform mode, a transform operation is performed before quantizing the residual block to be encoded currently.
[0065] In an embodiment of the present application, for multiple encoding of multimedia, in the first encoding stage of the multimedia multiple encoding process, the transformation mode information of the residual block is recorded; in the non-first encoding stage, the transformation mode information of each residual block recorded in the first encoding stage is obtained; based on the transformation mode information recorded in the first encoding stage and the attribute information of the current residual block to be encoded, the transformation mode information of the current residual block to be encoded is determined; the current residual block to be encoded is encoded according to the transformation mode information of the current residual block to be encoded; that is, the technical solution provided by the present application refers to the transformation mode information of each residual block in the first encoding, and determines the corresponding transformation mode information for the current residual block to be encoded in the non-first encoding stage, without the need to encode two modes at the same time and then select the optimal mode from the two modes, effectively reducing the computational complexity of the transformation mode information selection in multiple encoding, thereby reducing the code stream encoding time and computing resource overhead, and helping to improve coding efficiency.
[0066] It should be noted that, in other embodiments of the present application, in a non-first encoding stage, the transform mode information of the residual block to be encoded can be determined based on the transform mode information recorded in the previous encoding stage and the transform mode information of the residual block to be encoded currently; for example, in the second encoding stage, the transform mode information of the residual block to be encoded currently in the second encoding is determined based on the transform mode information recorded in the first encoding stage and the attribute information of the residual block to be encoded currently in the second encoding, and in the second encoding stage, the transform mode information of each residual block corresponding to the second encoding is recorded; in the third encoding stage, the transform mode information of the residual block to be encoded currently in the second encoding is determined based on the transform mode information recorded in the second encoding stage and the transform mode information of the residual block to be encoded currently in the third encoding, and so on.
[0067] In one embodiment of the present application, another multimedia encoding method is provided. The multimedia encoding method can be applied to the implementation environment shown in FIG1 . The method can be executed by an electronic device, for example, by a content producer or a server, or by both the content producer and the server. In the embodiment of the present application, the method is described by taking the server as an example. As shown in FIG4 , the multimedia encoding method is based on FIG2 , with step S210 expanded to S410 to S420, and step S220 expanded to S430. Steps S410 to S430 are described in detail as follows:
[0068] S410 : In a first encoding stage of multiple encodings of multimedia, obtain the size of each residual block and transform mode information of each residual block.
[0069] As described above, during the initial encoding process, after performing a prediction operation on multimedia, the size of each residual block can be obtained. The size of each residual block can be different, for example, 8×8 pixels, 16×16 pixels, etc. After performing operations such as transformation and quantization on the residual block, transform mode information for each residual block can be obtained.
[0070] S420 . According to the size of each residual block, save the transform mode information of each residual block to a corresponding flag bit.
[0071] In an embodiment of the present application, in order to facilitate recording the transform mode information of each residual block, the corresponding flag bit in the transform mode information is saved. The value of a flag bit can be considered as a flag value, which occupies one bit of storage space, for example, it can be stored in a storage resource such as a memory. There is a corresponding relationship between the residual block and the flag bit, and then the server determines the flag bit corresponding to the residual block based on the size of the residual block, and then saves the corresponding transform mode information to the flag bit. For example, if the residual block is transformed, the flag bit is marked as 1, and if the residual block is skipped, the flag bit is marked as 0.
[0072] It should be noted that different residual blocks have different corresponding flag bits, that is, the flag bits corresponding to different residual blocks do not overlap. In one example, there is a corresponding relationship between the size of the residual block and the number of flag bits corresponding to the residual block. A residual block can correspond to multiple flag bits, that is, the transform mode information of a residual block can be stored in multiple corresponding flag bits.
[0073] In one example, the flag storing the transformation mode information may be stored in a memory, or stored in a disk in the form of a read-write file.
[0074] S430 . In a non-initial encoding stage, obtain the transform mode information of each residual block recorded in the initial encoding stage from the flag bit.
[0075] In an embodiment of the present application, during non-initial encoding, the server can extract all flag bits from the memory and obtain the transformation mode information of each residual block recorded in the initial encoding stage from the content marked by each flag bit.
[0076] It should be noted that for other detailed descriptions of steps S230 to S240 shown in FIG4 , please refer to steps S230 to S240 shown in FIG2 , which will not be described again here.
[0077] In an embodiment of the present application, in the first encoding stage, the transformation mode information of each residual block is saved in the corresponding flag bit according to the size of each residual block, so as to facilitate storage and also facilitate subsequent acquisition of the flag bit corresponding to each residual block.
[0078] The present application provides another multimedia encoding method, which can be applied to the implementation environment shown in FIG1. The method can be executed by the content producer or the server, or by the content producer and the server. In the present application, the method is described as being executed by the server. As shown in FIG5, the multimedia encoding method is based on FIG4, and step S210 or S420 is expanded to steps S510 to S540. Steps S510 to S540 are described in detail as follows:
[0079] S510: Obtain the minimum size of the residual block.
[0080] In an embodiment of the present application, the minimum size of the residual block may be the size of the minimum residual block, that is, the predetermined minimum size of the residual block, for example, the minimum size of the residual block is 2×2 pixels. In other embodiments, the minimum size of the residual block may also be a size that is widely used in video coding and can provide sufficient information, for example, the minimum size of the residual block is generally 4×4 pixels. It should be noted that the sub-blocks mentioned in this application are sub-blocks into which the residual block is divided according to the minimum size.
[0081] S520: Determine a sub-block of a minimum size included in the residual block according to the size and the minimum size of the residual block.
[0082] In the embodiment of the present application, after the size and minimum size of the residual block are known, the minimum size sub-block contained in the residual block can be determined. For example, if the size of the residual block is 8×8 pixels and the minimum size is 2×2 pixels, then the residual block contains 16 2×2 pixel sub-blocks.
[0083] S530: Allocate a flag bit to each sub-block of the minimum size included in the residual block.
[0084] S540: Save the transform mode information of each sub-block to a corresponding flag bit to record the transform mode information of the corresponding residual block. The transform mode information of each residual block includes the transform mode information of each sub-block contained therein. Depending on the size of the residual block, a residual block may include one or more minimum-sized sub-blocks, and each minimum-sized sub-block may be associated with one or more flag bits. Each flag bit is used to represent the transform mode information of a sub-block in the residual block.
[0085] In an embodiment of the present application, there is a correspondence between the number of sub-blocks of the minimum size covered by the residual block and the number of flag bits corresponding to the residual block. The correspondence can be a one-to-one correspondence, that is, one sub-block of the minimum size corresponds to one flag bit; the correspondence can also be a one-to-many correspondence, for example, one residual block of the minimum size corresponds to 2 flag bits. The specific correspondence can be flexibly adjusted according to actual conditions and is not limited here.
[0086] The position of a sub-block of a residual block relative to other sub-blocks is the same as the position of the flag bit corresponding to the sub-block of the residual block relative to other flag bits; for example, if residual block A is located to the left of residual block B, then the flag bit corresponding to residual block A is equivalent to the flag bit corresponding to residual block B being located to the left.
[0087] In an embodiment of the present application, after determining the flag bit corresponding to the residual block, the transformation mode of the residual block is saved to the determined flag bit. For example, the residual block covers 16 2×2 pixels, and one 2×2 pixel residual block corresponds to one flag bit, then the transformation mode information of the residual block is saved to 16 flag bits. Assuming that the transformation mode information of the residual block is to skip the transformation operation, the flag bits at these 16 corresponding positions will be marked as no transformation operation is performed.
[0088] As shown in FIG6 , the black squares represent 2×2 pixel sub-blocks, which are included in the residual block 601. Assuming that the residual block 601 chooses not to perform the transformation operation, which corresponds to the flag bits of 16 sub-blocks, and the residual block 602 performs the transformation operation, which corresponds to the flag bits of 9 sub-blocks, then the 16 sub-blocks corresponding to the residual block 601 are marked in white, and the 9 sub-blocks corresponding to the residual block 602 are marked in shades to distinguish different choices of different residual blocks.
[0089] It should be noted that, for the detailed description of steps S410, S430, S230-S240 shown in FIG5, please refer to steps S410, S430, S230-S240 shown in FIG4, which will not be repeated here.
[0090] In the embodiment of the present application, based on the size of the residual block and the minimum size of the residual block, the number of sub-blocks of the minimum size contained in the residual block is calculated, and then the corresponding flag bits are determined to adapt to residual blocks of various sizes and ensure that the corresponding flag bits can be determined for residual blocks of various sizes.
[0091] The present application also provides another multimedia encoding method, which can be applied to the implementation environment shown in FIG1 . The method can be executed by the content producer or the server, or by both the content producer and the server. In the present application, the method is described by taking the server as an example. As shown in FIG7 , the multimedia encoding method is based on the method shown in FIG4 , but S230 is expanded to S710 to S730. Steps S710 to S730 are described in detail as follows:
[0092] S710. Obtain, based on the transform mode information of each residual block recorded in the first encoding stage and the size and position of the residual block currently to be encoded, a reference flag corresponding to each sub-block in the residual block currently to be encoded, wherein each sub-block has a minimum size of a residual block, and the reference flag corresponding to each sub-block in the residual block currently to be encoded is the flag of the sub-block at the same position in the first encoding stage. Here, the sub-block at the same position in the first encoding stage means that the position is the same as that of the corresponding sub-block in the residual block currently to be encoded, that is, the sub-blocks are in the same image area.
[0093] In one embodiment, S710 may include: determining a coverage area of the residual block to be encoded based on the size and position of the residual block to be encoded; and determining, based on the transform mode information of each residual block recorded during the initial encoding phase, a flag bit of a sub-block within the coverage area, as a reference flag bit corresponding to a sub-block at the same position in the residual block to be encoded. The coverage area refers to the image region where the residual block to be encoded is located.
[0094] S720. Determine, according to the reference flag corresponding to each sub-block in the current residual block to be encoded, the distribution of sub-blocks to which the transform skip mode is applied in the sub-blocks having the same position as the sub-block in the current residual block to be encoded in the first encoding stage.
[0095] In some embodiments, the transformation mode information of each residual block is recorded in the first encoding stage, including the number and position of the flag bits corresponding to the transformation mode information of each residual block, that is, the distribution of the transformation mode information of the sub-blocks in each residual block; then, according to the number and position of the flag bits corresponding to the transformation mode information of each residual block recorded in the first encoding stage, the distribution of each flag bit indicating the application of the transformation skip mode in the first encoding can be obtained, that is, the distribution of the sub-blocks to which the transformation skip mode is applied, and based on the size and position of the residual block to be encoded currently, the distribution of the flag bits indicating the application of the transformation skip mode in the first encoding corresponding to the residual block to be encoded currently can be known, that is, the distribution of the sub-blocks to which the transformation skip mode is applied in each sub-block with the same position as the sub-block in the residual block to be encoded currently in the first encoding stage, which may include, for example, the number and position of the flag bits of the sub-blocks to which the transformation skip mode is applied.
[0096] S730: Determine transform mode information of the current residual block to be encoded according to the obtained distribution of the sub-blocks to which the transform skip mode is applied.
[0097] In one example, the transform mode information of the residual block to be encoded can be determined by the distribution of the flag indicating the application of the transform skip mode in the first encoding corresponding to the residual block to be encoded.
[0098] In another example, the transform mode information of the current residual block to be encoded can also be determined by the number and position of the flag bits indicating the application of the transform skip mode in the first encoding corresponding to the current residual block to be encoded, and the position of the current residual block to be encoded.
[0099] It should be noted that for other detailed descriptions of steps S410 to S430 and S240 shown in FIG. 7 , please refer to steps S410 to S430 and S240 shown in FIG. 4 , which will not be repeated here.
[0100] In an embodiment of the present application, the distribution of the flag bits indicating the application of the transform skip mode in the first encoding corresponding to the current residual block to be encoded is obtained through the transform mode information recorded in the first encoding stage and the size and position of the current residual block to be encoded, so as to determine the impact of the distribution of the flag bits indicating the application of the transform skip mode in the first encoding on the current residual block to be encoded, thereby ensuring the accuracy of the transform mode information of the current residual block to be encoded.
[0101] In one embodiment of the present application, another multimedia encoding method is provided. The multimedia encoding method can be applied to the implementation environment shown in FIG1 . The method can be executed by a content producer or a server, or can be executed by both the content producer and the server. In the embodiment of the present application, the method executed by the server is used as an example for description. As shown in FIG8 , the multimedia encoding method is extended from S710 to S810 to S830 based on the method shown in FIG7 . Steps S810 to S830 are described in detail as follows:
[0102] In an embodiment of the present application, the transformation mode information of each residual block is recorded in the first encoding stage. In one example, the transformation mode information of each residual block is marked by a flag bit distribution map, as shown in FIG6 , and then the position of each flag bit that does not use the transformation operation can be determined from all flag bits through the change pattern distribution of the sub-blocks of the residual block (which can be considered as a flag bit distribution map), that is, the position distribution of the sub-blocks that do not use the transformation operation. The position of each flag bit that does not use the transformation operation refers to the position of the flag bit in the flag bit distribution map, which can be used to characterize the position distribution of the sub-blocks that do not use the transformation operation.
[0103] S810: Determine a coverage area of the current residual block to be encoded according to the size and position of the current residual block to be encoded.
[0104] It can be understood that the position of the flag corresponding to the transform mode information of the residual block in the flag distribution map matches the corresponding position of the residual block in the flag distribution map, and the area corresponding to the current residual block to be encoded can be determined through the size of the current residual block to be encoded. For example, an 8×8 pixel current residual block to be encoded has an area corresponding to 16 2×2 pixels in the flag distribution map; through the position of the current residual block to be encoded, it can be determined at which position in the flag distribution map the area corresponding to the current residual block to be encoded corresponds, and then the coverage area of the current residual block to be encoded in the flag distribution map is obtained.
[0105] S820. Determine, based on the transformation mode information of each residual block recorded in the first encoding stage, the flag of the sub-block in the coverage area in the first encoding stage, as a reference flag corresponding to the sub-block at the same position in the current residual block to be encoded.
[0106] In an embodiment of the present application, from the various flag bits indicating the application of the transform skip mode, a flag bit located within the coverage area of the current residual block to be encoded is selected, and then the flag bit indicating the application of the transform skip mode in the first encoding covered by the current residual block to be encoded is obtained, that is, the flag bit of the sub-block that is within the coverage area in the first encoding stage is determined.
[0107] It should be noted that for other detailed descriptions of steps S410 to S430, S720, and S240 shown in FIG8 , please refer to steps S410 to S430, S720, and S240 shown in FIG7 , which will not be repeated here.
[0108] In an embodiment of the present application, the coverage area of the residual block to be encoded is determined based on the size and position of the residual block to be encoded, and then the flag bits indicating the application of the transform skip mode located in the coverage area are selected to determine the impact of the distribution of the flag bits indicating the application of the transform skip mode in the first encoding on the residual block to be encoded, thereby ensuring the accuracy of the determination of the transform mode information of the residual block to be encoded.
[0109] The embodiment of the present application provides a multimedia encoding method, which includes an implementation environment that can be applied to the implementation environment shown in Figure 1. The method can be executed by the content producer or the server, or by the content producer and the server. In the embodiment of the present application, the method is executed by the server as an example for explanation. As shown in Figure 9, the multimedia encoding method is based on the method shown in Figure 7, and S720 is expanded to S910-S920. Steps S910-S920 are described in detail as follows:
[0110] S910: Determine, based on the obtained distribution of the sub-blocks to which the transform skip mode is applied and the size and position of the current residual block to be encoded, a ratio of the number of sub-blocks to which the transform skip mode is applied to the total number of sub-blocks included in the current residual block to be encoded. The total number of sub-blocks included in the current residual block to be encoded is the same as the number of sub-blocks within the coverage area of the current encoded residual block in the initial encoding.
[0111] In an embodiment of the present application, the area of the sub-block of the flag indicating the application of the transform skip mode in the first encoding corresponding to the current residual block to be encoded (i.e., the area of the sub-block of the transform skip mode in the first encoding) can be obtained through the distribution of the flag indicating the application of the transform skip mode in the first encoding covered by the current residual block to be encoded (the obtained distribution of the sub-blocks to which the transform skip mode is applied in the first encoding), and the area corresponding to the current residual block to be encoded can be obtained through the size and position of the current residual block to be encoded, and then the area corresponding to the current residual block to be encoded can be compared with the area corresponding to the current residual block to be encoded to determine the ratio of the number of sub-blocks to which the transform skip mode is applied to the total number of sub-blocks contained in the current residual block to be encoded.
[0112] S920: Determine the transform mode information of the current residual block to be encoded according to the ratio.
[0113] In an embodiment of the present application, the above-mentioned ratio can reflect the degree of influence of not using the transformation operation on the current residual block to be encoded. The larger the ratio, the greater the degree of influence of not using the transformation operation on the current residual block to be encoded, and the greater the probability that the transformation mode information of the current residual block to be encoded is to skip the transformation operation.
[0114] In one example, the corresponding transform mode information may be selected for the current residual block to be encoded by comparing the ratio with a preset proportion threshold.
[0115] It should be noted that for other detailed descriptions of steps S410 to S430, S710, and S240 shown in FIG9 , please refer to steps S410 to S430, S710, and S240 shown in FIG7 , which will not be repeated here.
[0116] In the embodiment of the present application, the ratio is used to reflect the degree of influence of not using the transformation operation on the current residual block to be encoded, and then the transformation mode information of the current residual block to be encoded is determined, thereby ensuring the accuracy and reliability of the determined transformation mode.
[0117] The present application provides another multimedia encoding method, which can be applied to the implementation environment shown in FIG1 . The method can be executed by a content producer or a server, or by both the content producer and the server. In the present application, the method is described by taking the server as an example. As shown in FIG10 , the multimedia encoding method extends FIG9 by extending S920 to S1010 to S1020. Steps S1010 to S1020 are described in detail as follows:
[0118] S1010: If the ratio is greater than a first preset ratio threshold, determine that the transform mode information of the current residual block to be encoded indicates application of a transform skip mode.
[0119] S1020. If the ratio is less than or equal to a second preset ratio threshold, determine that the transformation mode information of the current residual block to be encoded is a mode indicating that the application performs transformation, and the second preset ratio threshold is less than the first preset ratio threshold.
[0120] In an embodiment of the present application, based on the comparison relationship between the ratio and the preset proportion threshold, the corresponding transform mode information is selected for the current residual block to be encoded; wherein, if the ratio is greater than the first preset proportion threshold, it means that the current residual block to be encoded covers more flag bits indicating the application of the transform skip mode, which means that the image content in this area contains more high-frequency components after the prediction, and the possibility that the transform operation cannot have the effect of concentrating the residual energy of the residual block to be encoded is greater. Therefore, the transform mode information of the current residual block to be encoded is determined to indicate the transform skip mode.
[0121] If the ratio is a second preset proportion threshold, and the second preset proportion threshold is smaller than the first preset proportion threshold, it indicates that the flag indicating the application of the transform skip mode covered by the current residual block to be encoded is too small to affect the transform operation. Therefore, the transform mode information of the current residual block to be encoded is determined to be a mode indicating the execution of the transform.
[0122] In one example, the first preset proportion threshold can be flexibly adjusted according to actual conditions. For example, the first preset proportion threshold is 30%. In some scenarios, the first preset proportion threshold can also be adjusted according to the area of the residual block to be encoded. For example, a standard first preset proportion threshold is set. The standard first preset proportion threshold corresponds to the standard area of the residual block. The larger the area of the residual block to be encoded is, the larger the first preset proportion threshold is.
[0123] In one example, the second preset proportion threshold is 0, and in another example, the second preset proportion threshold is 2%. It can be flexibly adjusted according to actual conditions and is not limited here.
[0124] It should be noted that for other detailed descriptions of steps S410 to S430, S710, S910, and S240 shown in FIG10 , please refer to steps S410 to S430, S710, S910, and S240 shown in FIG9 , which will not be repeated here.
[0125] In an embodiment of the present application, if the ratio is greater than the first preset proportion threshold, it means that the current residual block to be encoded covers more flag bits indicating the application of the transform skip mode, and thus the transform mode information of the current residual block to be encoded is determined to indicate the transform skip mode. If the ratio is less than or equal to the second preset proportion threshold, the transform mode information of the current residual block to be encoded is determined to be a mode indicating the execution of the transform, thereby ensuring the rationality of the determination of the transform mode information.
[0126] The present application provides another multimedia encoding method, which includes an implementation environment that can be applied to the implementation environment shown in Figure 1. The method can be executed by the content producer or the server, or by the content producer and the server. In the present application, the method is executed by the server as an example for explanation. As shown in Figure 11, the multimedia encoding method is based on the method shown in Figure 10, and steps S1110 to S1120 are added. Steps S1110 to S1120 are described in detail as follows:
[0127] S1110: If the ratio is less than the first preset proportion threshold and greater than the second preset proportion threshold, obtain an area at a predetermined position of the current residual block to be encoded.
[0128] In an embodiment of the present application, if the ratio is less than the first preset ratio threshold and greater than the second preset ratio threshold, it means that the image content of the image area where the current residual block to be encoded is located contains neither too many nor too few high-frequency components after prediction. In order to accurately determine the transformation mode information of the current residual block to be encoded, the transformation mode information of the current residual block to be encoded is further determined based on the position of the aforementioned flag indicating the application of the transformation skip mode. At this time, the area of the predetermined position of the current residual block to be encoded will be obtained.
[0129] In one example, the area of the predetermined position of the current residual block to be encoded is the area corresponding to one quarter of the upper left corner of the residual block to be encoded. This is because according to the encoding rules of the quantization coefficients, the closer the transform coefficient with a large absolute value is to the upper left corner of the transform block, the fewer bits are required. The encoding order of the coefficients is scanned in a "Z" shape from the upper left corner to the lower right corner. The smaller coefficients are all 0 after quantization. It is only necessary to encode a subsequent all-0 flag after the non-zero coefficient encoding is completed. Therefore, the upper left corner of the residual block to be encoded has a transform coefficient with a large absolute value, and no transform operation may have better rate-distortion performance.
[0130] In one example, when determining the area corresponding to one quarter of the upper left corner of the residual block to be encoded, the residual block to be encoded is evenly divided into four parts to determine the area corresponding to the upper left corner. As shown in Figure 12A, the residual block to be encoded 1201 is evenly divided into four parts, and the area in the upper left corner is used as the predetermined position area 1202.
[0131] In another example, the area of the predetermined position of the current residual block to be encoded may be smaller than the area corresponding to one quarter of the upper left corner of the residual block to be encoded; as shown in FIG12B , the area 1203 of the predetermined position of the residual block to be encoded 1201 is smaller than the area 1202. The specific range of the area of the predetermined position can be flexibly adjusted according to actual conditions and is not limited here.
[0132] S1120 : Determine transform mode information of the current residual block to be encoded according to the relationship between the position of the sub-block to which the transform skip mode is applied and the area of the predetermined position.
[0133] In an embodiment of the present application, the relationship between the position of the flag indicating the application of the transform skip mode and the area of the predetermined position refers to whether the position of the flag indicating the application of the transform skip mode is located within the area of the predetermined position, so as to determine whether the residual block to be encoded has a transform coefficient with a large absolute value, and then determine the corresponding transform mode information for the current residual block to be encoded.
[0134] It should be noted that for other detailed introductions of steps S410 to S430, S710, S910, S1010 to S1020, and S240 shown in Figure 11, please refer to steps S410 to S430, S710, S910, S1010 to S1020, and S240 shown in Figure 10, which will not be repeated here.
[0135] In an embodiment of the present application, if the proportion is less than the first preset proportion threshold and greater than the second preset proportion threshold, the transform mode information of the current residual block to be encoded is determined based on the relationship between the position of the flag indicating the application of the transform skip mode and the area of the predetermined position, thereby further improving the reliability of the determination of the transform mode information.
[0136] It is worth noting that the embodiment of the present application provides another multimedia encoding method, which can be applied to the implementation environment shown in Figure 1. The method can be executed by the content producer or the server, or can be executed by the content producer and the server together. In the embodiment of the present application, the method is executed by the server as an example. As shown in Figure 13, based on the method shown in Figure 11, S1120 is expanded to step S1310, which is described in detail as follows:
[0137] S1310: If the positions of the sub-blocks to which the transform skip mode is applied are all within the area of the predetermined position, determine that the transform mode information of the current residual block to be encoded indicates application of the transform skip mode.
[0138] In an embodiment of the present application, if the positions of the various flag bits indicating the application of the transform skip mode are all within the predetermined position area, it indicates that the residual block to be encoded has the characteristic that coefficients with large absolute values are concentrated in the upper left corner, and better rate-distortion performance may be achieved without performing the transform. Therefore, the transform mode information of the current residual block to be encoded is determined to be a skip transform operation.
[0139] It should be noted that for other detailed introductions of steps S410 to S430, S710, S910, S1010 to S1020, S1110, and S240 shown in Figure 13, please refer to steps S410 to S430, S710, S910, S1010 to S1020, S1110, and S240 shown in Figure 11, which will not be repeated here.
[0140] The embodiment of the present application provides another multimedia encoding method. The multimedia encoding method includes an implementation environment that can be applied to the implementation environment shown in FIG1 . The method can be executed by a content producer or a server, or can be executed by the content producer and the server together. In the embodiment of the present application, the method executed by the server is used as an example for explanation. As shown in FIG14 , the multimedia encoding method expands S1120 to steps S1410 to S1420 based on the method shown in FIG11 . The details are as follows:
[0141] S1410. If the position of at least one sub-block to which the transform skip mode is applied is outside the area of the predetermined position, determine that the transform mode information of the current residual block to be encoded indicates selecting the optimal mode from the transform execution mode and the transform skip mode.
[0142] S1420: Select an optimal mode from the transform mode and the transform skip mode. In one embodiment, the embodiment of the present application can calculate the coding rate distortion costs of the transform mode and the transform skip mode, and then select an optimal mode from the transform mode and the transform skip mode based on the coding rate distortion costs of the transform mode and the transform skip mode.
[0143] In an embodiment of the present application, if there is at least one flag indicating the application of the transform skip mode, the position of which is outside the area of the predetermined position, indicating that the flag indicating the application of the transform skip mode is relatively scattered. At this time, the server may not make a quick judgment, but may choose to use the transform operation and skip the transform operation for the current residual block to be encoded, that is, try the methods of performing the transform and skipping the transform.
[0144] To ensure the reliability of the transform mode selection for the current residual block, the optimal encoding operation is selected based on the rate-distortion cost, either using the transform operation or skipping the transform operation. The calculated rate-distortion cost refers to the number of bits required to achieve a certain image quality. When selecting the most appropriate mode, the encoder compares the calculated rate-distortion costs of the two modes and selects the mode with the lower cost for encoding.
[0145] In one example, the server can also determine which encoder and decoder to use to implement compression and decompression, determine the range of bit rates based on the type of data to be compressed and the required compression rate, and then calculate the distortion cost at each bit rate for each bit rate; the distortion cost refers to the degree of difference between the compressed image and the original image. The distortion costs at each bit rate in the two modes are compared, and the smallest distortion cost is selected as the most appropriate mode.
[0146] In other embodiments of the present application, if there is a flag bit indicating the application of the transform skip mode and its position is outside the area of the predetermined position, and there is also a flag bit indicating the application of the transform skip mode and its position is within the area of the predetermined position, and the number of flag bits indicating the application of the transform skip mode within the area of the predetermined position and outside the area of the predetermined position is equal, that is, the difference in number is not greater than the preset difference threshold, then the transform mode information of the current residual block to be encoded is determined to be using transform operation and skipping transform operation, and then the optimal encoding operation mode is selected from using transform operation and skipping transform operation through rate-distortion cost.
[0147] In other embodiments of the present application, if the positions of the flag bits indicating the application of the transform skip mode are all outside the predetermined position area, the transform mode information of the current residual block to be encoded can be directly determined to use the transform operation without selecting through two operations.
[0148] It should be noted that for other detailed introductions of steps S410 to S430, S710, S910, S1010 to S1020, S1110, and S240 shown in Figure 14, please refer to steps S410 to S430, S710, S910, S1010 to S1020, S1110, and S240 shown in Figure 11, which will not be repeated here.
[0149] In the embodiment of the present application, for the area where the flag indicating the application of the transform skip mode is located outside the predetermined position, different transform mode information is selected as the optimal mode, which can further improve the reliability of the determination of the transform mode information.
[0150] Please refer to FIG. 15 , which is a flowchart of a multimedia encoding method according to an embodiment of the present application. The method can be applied to the implementation environment shown in FIG. 1 . The method can be executed by a content producer or a server, or can be executed by both the content producer and the server. In the embodiment of the present application, the method executed by the server is used as an example for description. The multimedia encoding method expands S910 of FIG. 9 into S1510 to S1520, which are described in detail as follows:
[0151] S1510 : Calculate the total number of sub-blocks of the minimum size included in the residual block to be encoded currently according to the size of the residual block to be encoded currently and the minimum size of the residual block.
[0152] In an embodiment of the present application, in order to accurately calculate the proportion of the sub-block corresponding to the flag indicating the application of the transform skip mode in the current residual block to be encoded, the current residual block to be encoded is split into multiple sub-blocks of minimum size according to the size of the current residual block to be encoded and the minimum size of the residual block, and then the total number of sub-blocks of minimum size contained in the current residual block to be encoded is calculated.
[0153] S1520: Calculate a ratio of the number of sub-blocks to which the transform skip mode is applied to the total number.
[0154] In an embodiment of the present application, the ratio obtained by dividing the number of sub-blocks to which the transform skip mode is applied by the total number of sub-blocks of the minimum size contained in the current residual block to be encoded is used as the proportion of the sub-blocks corresponding to the flag indicating the application of the transform skip mode in the current residual block to be encoded.
[0155] It should be noted that for other detailed descriptions of steps S410 to S430, S710, S920, and S240 shown in FIG15 , please refer to steps S410 to S430, S710, S920, and S240 shown in FIG9 , which will not be repeated here.
[0156] In an embodiment of the present application, based on the size of the current residual block to be encoded and the minimum size of the residual block, the number of corresponding flag bits of the current residual block to be encoded can be obtained. Combined with the number of flag bits indicating the application of the transform skip mode in the first encoding covered by the current residual block to be encoded, the proportion of the sub-blocks corresponding to the flag bits indicating the application of the transform skip mode in the current residual block to be encoded is calculated.
[0157] For ease of understanding, the multimedia encoding method provided in the embodiment of the present application is described in detail below based on the implementation environment shown in FIG1 , taking the multimedia as video as an example:
[0158] In the multimedia encoding method provided in this embodiment, the video is encoded multiple times to obtain higher bit rate accuracy and better coding rate distortion performance. When the video is encoded for the first time, the encoder performs main steps such as prediction, transformation, quantization, and entropy encoding on the video to obtain a compressed binary code stream.
[0159] It is worth noting that when the first encoding is performed, the transform mode information of each residual block will be saved, that is, whether it uses a transform operation. Since the block division results of each encoded image may be different, in order to be able to utilize the transform mode information of the residual block at different divisions, the embodiment of the present application saves the transform mode information of each residual block in a 2×2 block size; this is because the minimum size of the residual block will not be less than 2×2 pixels. Although the residual block can have a variety of sizes, it can always be divided into several 2×2 sub-blocks. For ease of storage, in the embodiment of the present application, each 2×2 residual block corresponds to a flag bit. For example, for a frame of an image with a size of 1920×1080, 860×540 flag bits need to be saved, and the flag bits can be saved in the memory or saved in the disk system in the form of a read-write file.
[0160] Therefore, during the first encoding, information about whether the transform operation is performed on each residual block in the final partitioning form will be recorded in its corresponding flag. For example, if an 8×8 residual block chooses not to perform the transform operation, since it covers 16 2×2 sub-blocks, its corresponding 16 flag bits will all be marked as not performing the transform operation.
[0161] When performing subsequent multiple encodings, such as the second encoding, the transformation mode information saved in the first encoding pass will be used as an important basis for judging whether the current residual block to be encoded needs to be transformed in the second encoding. For the current residual block to be encoded, the embodiment of the present application will first obtain the transformation mode information saved in the first encoding pass, and count the proportion and position of the 2×2 blocks that skip the transformation. As shown in Figure 16, the black squares represent the flag bits that store whether the transformation operation is used for each 2×2 block. There are 16×16 flag bits saved in the first encoding, of which the 2×2 blocks that do not use the transformation operation are marked as shaded, and the 2×2 blocks that use the transformation operation are white.
[0162] It's worth noting that whether a residual block to be coded undergoes a transform operation is determined by the proportion of 2×2 blocks it covers that were not transformed in the first encoding pass, as well as the location distribution of these 2×2 blocks. As shown in Figure 16, 1601 is the transform mode information recorded during the first encoding phase, area 1602 is the 16×16 residual block to be coded, and its predetermined location area 1603 is circled with a dashed box. This residual block to be coded covers 20 2×2 blocks that were not transformed (the shaded area in the figure), accounting for 31.25% of all the 2×2 blocks it covers. All of these 2×2 blocks that were not transformed are outside the predetermined location area.
[0163] Generally speaking, the more 2×2 blocks that skip transform operations are covered in an area, the more high-frequency components the residual of the image content in this area contains after prediction, and the greater the possibility that the transform will not have the effect of concentrating the residual energy of this residual block. At the same time, according to the coding rules of the quantization coefficients, the closer the transform coefficient with large absolute value is to the upper left corner of the transform block, the fewer bits are required. This is because the coding order of the coefficients is scanned from the upper left corner to the lower right corner in a "Z" shape. The smaller coefficients are all 0 after quantization. It is only necessary to encode a subsequent all-0 flag after the non-zero coefficients are encoded. When the 2×2 blocks that skip transform all fall in the area of the predetermined position of the residual block to be encoded, the residual block to be encoded may already have the characteristic that the coefficients with large absolute values are concentrated in the upper left corner of the block, and not performing the transform may have better rate-distortion performance. Therefore, the transform mode information of the current residual block to be encoded can be determined through the following four situations:
[0164] a) When the proportion of the 2×2 blocks skipped in the residual block to be coded is greater than k, the residual block to be coded is not transformed and is directly quantized.
[0165] b) When the residual block to be coded does not include a 2×2 block for which a transform operation is skipped, that is, k is 0, the residual block to be coded is only transformed and the transform skip mode is not attempted.
[0166] c) When the proportion of 2×2 blocks skipped in the residual block to be encoded is between 0 and k, and there are 2×2 blocks in the area of the residual block to be encoded that is not in the predetermined position, the encoder does not make a quick judgment, but tries both the transformation and non-transformation methods, and selects the best method based on the rate-distortion.
[0167] d) When the proportion of 2×2 blocks skipped in the residual block to be encoded is between 0 and k, and all 2×2 blocks are located in the area of the predetermined position of the residual block to be encoded, the residual block to be encoded is coded without transform, that is, the transform is skipped and quantization is performed directly.
[0168] In one example, k is set to 20% to achieve a better acceleration-loss ratio, and may also be set to other values according to specific requirements.
[0169] Based on the above content, an embodiment of the present application further provides a multimedia encoding method, as shown in FIG17 , the method comprising:
[0170] S1701: For the second encoding stage, obtain the current residual block to be encoded.
[0171] S1702: Obtain transform mode information of a 2×2 sub-block corresponding to the current residual block to be encoded during the first encoding.
[0172] S1703: Count the proportion of 2×2 sub-blocks that skip the transformation operation.
[0173] S1704 : Compare the proportion with a preset proportion threshold to determine whether the proportion is greater than the preset proportion threshold. If so, go to S1705 ; if not, go to S1706 .
[0174] S1705: Perform a non-transform coding operation on the current residual block to be coded.
[0175] S1706: Determine whether the proportion is greater than 0 and the 2×2 areas to be skipped are both in the area of the predetermined position of the current residual block to be encoded. If so, go to S1705; if not, go to S1707.
[0176] S1707: Perform a transform coding operation on the current residual block to be coded.
[0177] In the embodiment of the present application, when it is determined that the proportion is equal to 0, a transform coding operation is performed on the current residual block to be encoded.
[0178] In an embodiment of the present application, using the transformation mode information of the first encoding in multiple encodings to accelerate the selection process of the transformation mode information of subsequent encodings can reduce the computational complexity, thereby reducing the bitstream encoding time and computing resource overhead, and help improve encoding efficiency.
[0179] The present invention introduces an apparatus embodiment that can be used to execute the multimedia encoding method in the above-mentioned embodiment of the present invention. For details not disclosed in the apparatus embodiment of the present invention, please refer to the above-mentioned embodiment of the multimedia encoding method of the present invention.
[0180] The embodiment of the present application provides a multimedia encoding device, as shown in FIG18 . The multimedia encoding device can be used in a content producer or a server, and includes:
[0181] The recording module 1810 is configured to record the transform mode information of the residual block during the first encoding phase of the multimedia multiple encoding process;
[0182] An acquisition module 1820 is configured to acquire, in a non-initial encoding stage, transform mode information of each residual block recorded in the initial encoding stage;
[0183] a determination module 1830 configured to determine the transform mode information of the residual block to be encoded currently based on the transform mode information recorded in the first encoding stage and the attribute information of the residual block to be encoded currently;
[0184] The encoding module 1840 is configured to encode the current residual block to be encoded according to the transform mode information of the current residual block to be encoded.
[0185] It should be noted that the apparatus provided in the above embodiment and the method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs operations has been described in detail in the method embodiment and will not be repeated here.
[0186] An embodiment of the present application also provides an electronic device, comprising one or more processors and a storage device, wherein the storage device is used to store one or more computer programs, and when the one or more computer programs are executed by one or more processors, the electronic device implements the multimedia encoding method as described above.
[0187] FIG19 shows a schematic structural diagram of a computer system suitable for implementing an electronic device according to an embodiment of the present application.
[0188] It should be noted that the computer system 1900 of the electronic device shown in FIG19 is merely an example and should not impose any limitations on the functions and scope of use of the embodiments of the present application, wherein the electronic device may be a client or a server.
[0189] As shown in Figure 19, computer system 1900 includes a processor (Central Processing Unit, CPU) 1901, which can perform various appropriate actions and processes according to the program stored in read-only memory (Read-Only Memory, ROM) 1902 or the program loaded from storage part 1908 into random access memory (Random Access Memory, RAM) 1903, such as executing the method in the above embodiment. Various programs and data required for system operation are also stored in RAM 1903. CPU 1901, ROM 1902 and RAM 1903 are connected to each other via bus 1904. Input / output (I / O) interface 1905 is also connected to bus 1904.
[0190] In some embodiments, the following components are connected to the I / O interface 1905: an input section 1906 including a keyboard, a mouse, and the like; an output section 1907 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and speakers; a storage section 1908 including a hard disk; and a communication section 1909 including a network interface card such as a LAN (Local Area Network) card or a modem. The communication section 1909 performs communication processing via a network such as the Internet. A drive 1910 is also connected to the I / O interface 1905 as needed. Removable media 1911, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 1910 as needed, so that computer programs read from the removable media can be installed in the storage section 1908 as needed.
[0191] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 1909, and / or installed from a removable medium 1911. When the computer program is executed by the processor (CPU) 1901, the various functions defined in the system of the present application are executed.
[0192] It should be noted that the computer-readable medium shown in the embodiment of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (Erasable Programmable Read Only Memory), a flash memory, an optical fiber, a portable compact disk read-only memory (Compact Disc Read-Only Memory, CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, wherein a computer-readable computer program is carried. This propagated data signal can take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. A computer program embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, or any suitable combination thereof.
[0193] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions and operations of the devices, methods and computer program products according to various embodiments of the present application. Among them, each box in the flowchart or block diagram can represent a module, program segment, or part of the code, and the above-mentioned module, program segment, or part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and a computer program.
[0194] The units or modules described in the embodiments of the present application may be implemented in software or hardware, and the units or modules described may also be provided in a processor. The names of these units or modules do not, in certain circumstances, limit the units or modules themselves.
[0195] Another aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the aforementioned method. The computer-readable storage medium may be included in the electronic device described in the above embodiments, or may exist independently without being incorporated into the electronic device.
[0196] Another aspect of the present application further provides a computer program product, the computer program product including a computer program stored in a computer-readable storage medium. A processor of an electronic device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, causing the electronic device to perform the methods provided in the above-mentioned embodiments.
[0197] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiment of the application, the features and functions of two or more modules or units described above can be concretized in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.
[0198] Other embodiments of the present invention will readily occur to those skilled in the art after considering the specification and practicing the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of this application and include common knowledge or customary techniques in the art that are not disclosed herein.
[0199] The above content is only a preferred exemplary embodiment of the present application and is not intended to limit the implementation scheme of the present application. Ordinary technicians in this field can easily make corresponding changes or modifications based on the main concept and spirit of the present application. Therefore, the scope of protection of the present application shall be based on the scope of protection required by the claims.
[0200] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.
Claims
1. A multimedia encoding method, executed in an electronic device, the method comprising: In a first encoding stage of multiple encodings of multimedia, recording transform mode information of each residual block, wherein the transform mode information indicates whether a transform skip mode is applied; In a non-initial encoding stage, obtaining transform mode information of each residual block recorded in the initial encoding stage; Determining the transform mode information of the current residual block to be encoded according to the transform mode information recorded in the first encoding stage and the attribute information of the current residual block to be encoded; The current residual block to be encoded is encoded according to the transformation mode information of the current residual block to be encoded.
2. The method according to claim 1, wherein recording the transformation mode information of each residual block comprises: In the first encoding stage, the size of each residual block and the transform mode information of each residual block are obtained; According to the size of each residual block, saving the transformation mode information of each residual block to a corresponding flag bit; The step of obtaining the transform mode information of each residual block recorded in the first encoding stage in the non-first encoding stage includes: In the non-initial encoding stage, the transform mode information of each residual block recorded in the initial encoding stage is obtained from the flag bit.
3. The method according to claim 1 or 2, wherein recording the transformation mode information of each residual block comprises: Get the minimum size of the residual block; Determine, according to the size of each residual block and the minimum size, a sub-block of the minimum size included in the residual block; Allocating a flag bit to each sub-block of a minimum size included in the residual block; The transform mode information of each sub-block is saved to a corresponding flag bit to record the transform mode information of the corresponding residual block. The transform mode information of each residual block includes the transform mode information of each sub-block contained therein.
4. The method according to any one of claims 1 to 3, wherein the attribute information includes size and position; and determining the transform mode information of the current residual block to be encoded according to the transform mode information recorded in the first encoding stage and the attribute information of the current residual block to be encoded comprises: According to the transformation mode information of each residual block recorded in the first encoding stage and the size and position of the residual block to be encoded currently, obtain a reference flag corresponding to each sub-block in the residual block to be encoded currently, wherein each sub-block has a minimum size of the residual block, and the reference flag corresponding to each sub-block in the residual block to be encoded currently is a flag of a sub-block at the same position in the first encoding stage; Determine, according to the reference flag corresponding to each sub-block in the current residual block to be encoded, the distribution of sub-blocks to which the transform skip mode is applied in each sub-block having the same position as the sub-block in the current residual block to be encoded in the first encoding stage; According to the obtained distribution of the sub-blocks to which the transform skip mode is applied, transform mode information of the current residual block to be encoded is determined.
5. The method according to claim 4, wherein obtaining, according to the transform mode information of each residual block recorded in the first encoding stage and the size and position of the residual block to be encoded currently, a reference flag corresponding to each sub-block in the residual block to be encoded currently, wherein each sub-block has a minimum size of the residual block, and the reference flag corresponding to each sub-block in the residual block to be encoded currently is a flag of a sub-block at the same position in the first encoding stage, comprises: Determining a coverage area of the current to-be-encoded residual block according to the size and position of the current to-be-encoded residual block; According to the transformation mode information of each residual block recorded in the first encoding stage, the flag of the sub-block in the coverage area in the first encoding stage is determined as a reference flag corresponding to the sub-block at the same position in the current residual block to be encoded.
6. The method according to claim 4 or 5, wherein determining the transform mode information of the current residual block to be encoded according to the obtained distribution of the sub-blocks to which the transform skip mode is applied comprises: Determine, according to the obtained distribution of the sub-blocks to which the transform skip mode is applied and the size and position of the current residual block to be encoded, a ratio of the number of sub-blocks to which the transform skip mode is applied to the total number of sub-blocks included in the current residual block to be encoded; According to the ratio, transform mode information of the current residual block to be encoded is determined.
7. The method according to claim 6, wherein determining the transform mode information of the current residual block to be encoded according to the ratio comprises: If the ratio is greater than a first preset ratio threshold, determining that the transform mode information of the current residual block to be encoded indicates application of a transform skip mode; If the ratio is less than or equal to the second preset ratio threshold, it is determined that the transformation mode information of the current residual block to be encoded is an indication In the mode of applying the transformation, the second preset proportion threshold is less than the first preset proportion threshold.
8. The method according to claim 7, wherein determining the transform mode information of the current residual block to be encoded according to the ratio further comprises: If the proportion is less than the first preset proportion threshold and greater than the second preset proportion threshold, obtaining an area of a predetermined position of the current residual block to be encoded; The transform mode information of the current residual block to be encoded is determined according to the relationship between the position of the sub-block to which the transform skip mode is applied and the area of the predetermined position.
9. The method according to claim 8, wherein determining the transform mode information of the current residual block to be encoded according to the relationship between the position of the sub-block to which the transform skip mode is applied and the area of the predetermined position comprises: If the positions of the sub-blocks to which the transform skip mode is applied are all within the region of the predetermined position, it is determined that the transform mode information of the current residual block to be encoded indicates the application of the transform skip mode.
10. The method according to claim 8, wherein determining the transform mode information of the current residual block to be encoded according to the relationship between the position of the sub-block to which the transform skip mode is applied and the area of the predetermined position comprises: If the position of at least one sub-block to which the transform skip mode is applied is outside the area of the predetermined position, determining the transform mode information of the current residual block to be encoded to indicate selecting an optimal mode from a mode for performing transform and a transform skip mode; An optimum mode is selected from a mode in which transformation is performed and a transform skip mode.
11. The method according to claim 6, wherein determining the ratio of the number of sub-blocks to which the transform skip mode is applied to the current residual block to be encoded based on the obtained distribution of the sub-blocks to which the transform skip mode is applied and the size and position of the current residual block to be encoded comprises: Calculate the total number of sub-blocks of the minimum size included in the current residual block to be encoded according to the size of the current residual block to be encoded and the minimum size of the residual block; A ratio of the number of sub-blocks to which the transform skip mode is applied to the total number is calculated.
12. The method according to any one of claims 1 to 11, wherein encoding the current residual block to be encoded according to the transformation mode information of the current residual block to be encoded comprises: If the transform mode information of the current residual block to be encoded indicates to apply a skip transform mode, performing a quantization operation on the current residual block to be encoded; If the transformation mode information of the current residual block to be encoded indicates a mode in which transformation is to be applied, a transformation operation is performed before quantizing the current residual block to be encoded.
13. A multimedia encoding device, comprising: A recording module, configured to record transform mode information of each residual block in a first encoding stage of multiple encodings of multimedia, wherein the transform mode information indicates whether a transform skip mode is applied; An acquisition module, used for acquiring, in a non-first encoding stage, transform mode information of each residual block recorded in the first encoding stage; A determination module, configured to determine the transformation mode information of the current residual block to be encoded according to the transformation mode information recorded in the first encoding stage and the attribute information of the current residual block to be encoded; The encoding module is used to encode the current residual block to be encoded according to the transformation mode information of the current residual block to be encoded.
14. An electronic device comprising: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the electronic device to execute the method according to any one of claims 1 to 12. 15 . A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor of an electronic device, the electronic device executes the method according to claim 1 .
16. A computer program product, comprising a computer program, wherein the computer program is stored in a computer-readable storage medium, and a processor of an electronic device reads and executes the computer program from the computer-readable storage medium, so that the electronic device executes the method according to any one of claims 1 to 12.
Citation Information
Patent Citations
Code rate controlling method for video coding based on Rho domain
CN101287123A
Multi-transform-kernel fast skipping algorithm based on pre-selected layer
CN111355955A
Method and apparatus for video coding, computer device and storage medium
CN111988617A
Preencoder assisted video encoding
US20150350686A1
Method and apparatus for encoding a picture
WO2018096067A1
