Video encoding method, device, electronic device and storage medium

Abstract

This application discloses a video encoding method, apparatus, electronic device, and storage medium. The method includes: performing a first encoding process on a video to be encoded to obtain first bitstream data; extracting depth information from the first bitstream data to obtain first depth map data; inputting the first depth map data into a depth prediction model to obtain second depth map data; determining encoding information for a second encoding process based on the second depth map data, wherein the bitrate consumption of the second encoding process is lower than that of the first encoding process; and performing the second encoding process on the video to be encoded based on the encoding information to obtain an encoded video. Using this method, the bitrate consumption of video encoding can be reduced, thereby significantly reducing video encoding costs.

Description

Technical Field

[0001] The present invention relates to the technical field of video coding, and in particular to a video coding method, device, electronic equipment and storage medium. Background Art

[0002] Videos typically require encoding for end-to-end transmission. Currently, the H.265 encoder is the primary method for encoding videos. The H.265 encoder has high bitrate requirements, which incurs high bandwidth costs in large-scale deployment scenarios. To address this, the industry has developed a new generation of AV1 encoder, which can reduce bitrate overhead by over 20% compared to the H.265 encoder, significantly reducing bandwidth costs.

[0003] However, while the AV1 video encoder brings bandwidth savings, it also introduces more complex encoding tools. This results in a significant increase in encoding time, almost 50 times that of the H.265 encoder. This encoding time significantly increases computing power consumption, and thus increases computing costs. Summary of the Invention

[0004] In order to solve the above-mentioned problems existing in the prior art, the embodiments of the present application provide a video encoding method, device, electronic device and storage medium, which can reduce bandwidth expenses while alleviating the demand for computing power costs.

[0005] In a first aspect, an embodiment of the present application provides a video encoding method, the method comprising:

[0006] Performing a first encoding process on the video to be encoded to obtain first code stream data;

[0007] Extracting depth information from the first bitstream data to obtain first depth map data;

[0008] Inputting the first depth map data into a depth prediction model to obtain second depth map data;

[0009] Determining encoding information for a second encoding process based on the second depth map data, wherein the bit rate consumption of the second encoding process is lower than that of the first encoding process, and the computing power consumption of the first encoding process is lower than that of the second encoding process;

[0010] A second encoding process is performed on the video to be encoded according to the encoding information to obtain an encoded video.

[0011] In one possible implementation, inputting the first depth map data into a depth prediction model to obtain the second depth map data includes:

[0012] For the depth value corresponding to the coordinate (x, y) in the second depth map data, the depth value in the area (x–a, ya)-(x+a, y+a) in the first depth map data is input into the depth prediction model to obtain the depth value corresponding to the coordinate (x, y), where a is a natural number greater than or equal to 3;

[0013] All coordinates in the second depth map data are traversed to obtain the second depth map data.

[0014] In a possible implementation, before inputting the first depth map data into the depth prediction model to obtain the second depth map data, the method further includes:

[0015] Extending and assigning values ​​to a boundary area of ​​the first depth map data to obtain third depth map data;

[0016] The first depth map data is updated to the third depth map data.

[0017] In a possible implementation, determining encoding information for the second encoding process according to the second depth map data includes:

[0018] Determining a maximum coding unit size of the video to be encoded according to resolution information of the video to be encoded;

[0019] Determining a corresponding depth data region in the second depth map data according to the maximum coding unit size;

[0020] Count the modes of the depth values ​​in the depth data area;

[0021] Arrange the modes of the depth values ​​in descending order, and extract the depth values ​​corresponding to the first two modes of the arranged modes to obtain the first depth value and the second depth value;

[0022] A pre-established encoding relationship correspondence table is searched according to the first depth value and the second depth value to determine encoding information for the second encoding process.

[0023] In one possible implementation, the method further includes:

[0024] Obtaining multiple training video sets, wherein each training video set includes at least one training video, first reference depth map data of each training video in the at least one training video, and second reference depth map data of each training video, wherein the first reference depth map data is obtained after each training video is subjected to a first encoding process, and the second reference depth map data is obtained after each training video is subjected to a second encoding process;

[0025] Calling multiple prediction classifiers in the initial classification model to perform prediction processing on the first reference depth map data of each training video to obtain prediction results for each training video;

[0026] The initial classification model is trained based on the prediction results of each training video and the second reference depth map data of each training video to obtain a depth prediction model.

[0027] In a possible implementation, multiple prediction classifiers in the initial classification model are called to perform prediction processing on the first reference depth map data of each training video to obtain prediction results for each training video, including:

[0028] Traversing each prediction classifier in the initial classification model, based on the prediction results of each training video in the training video set corresponding to the last traversed prediction classifier obtained by processing the last traversed prediction classifier, determining the target training video set corresponding to the currently traversed target prediction classifier from multiple training video sets;

[0029] The target prediction classifier is called to perform prediction processing on the first reference depth map data of each training video in the target training video set to obtain a prediction result of each training video in the target training video set.

[0030] In one possible implementation, traversing each prediction classifier in the initial classification model, determining a target training video set corresponding to a currently traversed target prediction classifier from multiple training video sets based on prediction results of each training video in the training video set corresponding to the prediction classifier of the previous traversal obtained by processing the prediction classifier of the previous traversal, includes:

[0031] When the target prediction classifier is the first traversed prediction classifier, any one training video set is selected from multiple training video sets as the target training video set corresponding to the target prediction classifier;

[0032] When the target prediction classifier is not the first traversed prediction classifier, any one of the training videos whose prediction results in the target training video set of the last traversed prediction classifier are different from the second reference depth data corresponding to the target training video set, and the training video sets that did not participate in the training in multiple training video sets is selected as the target training video set corresponding to the target prediction classifier.

[0033] In a second aspect, an embodiment of the present application provides a video encoding device, including:

[0034] A first encoding module is configured to perform a first encoding process on the video to be encoded to obtain first bitstream data, and extract depth information from the first bitstream data to obtain first depth map data;

[0035] A prediction module, configured to input the first depth map data into a depth prediction model to obtain second depth map data;

[0036] The second encoding module is used to determine encoding information for a second encoding process based on the second depth map data, wherein the bit rate consumption of the second encoding process is lower than that of the first encoding process, and perform the second encoding process on the video to be encoded based on the encoding information to obtain an encoded video.

[0037] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, the processor being connected to a memory, the memory being used to store computer programs, and the processor being used to execute the computer programs stored in the memory, so that the electronic device performs the method of the first aspect.

[0038] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program enables a computer to execute the method of the first aspect.

[0039] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and is computer-operable to enable the computer to execute the method of the first aspect.

[0040] The implementation of the present application has the following beneficial effects:

[0041] In an embodiment of the present application, the first bitstream data of the video to be encoded is obtained through a first encoding process with low computing power consumption, and then the depth information is extracted from the first bitstream data to obtain the first depth map data of the video to be encoded. The first depth map data is then input into a pre-trained depth prediction model to obtain second depth map data for performing a second encoding process with low bit rate consumption. Finally, the corresponding encoding information is obtained based on the second depth map data, and the second encoding process is performed on the video to be encoded to obtain the encoded video. Thus, by predicting the first depth map data through the depth prediction model, the first encoding process with low computing power consumption is used to replace the second encoding process with high computing power consumption to perform the depth map data acquisition step, thereby reducing the computing power requirement after replacing the second encoding process. At the same time, using the second encoding process with low bit rate consumption to complete the subsequent video encoding steps can save a lot of bit rate compared to the first encoding process. This achieves the goal of reducing bandwidth expenses while smoothing the demand for computing power costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0043] Figure 1A schematic diagram of the hardware structure of a video encoding device provided in an embodiment of the present application;

[0044] Figure 2 A schematic diagram of a video encoding method according to an embodiment of the present invention;

[0045] Figure 3 A schematic diagram of depth map data corresponding to a coding unit provided in an embodiment of the present application;

[0046] Figure 4 A block diagram of the functional modules of a video encoding device provided in an embodiment of the present application;

[0047] Figure 5 A schematic structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0049] The terms "first," "second," "third," and "fourth," etc., in the specification and claims of this application and the accompanying drawings are used to distinguish between different objects, rather than to describe a specific order. In addition, the terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements, but may optionally include steps or elements not listed, or may optionally include other steps or elements inherent to the process, method, product, or apparatus.

[0050] Reference herein to an "embodiment" means that a particular feature, result, or characteristic described in conjunction with the embodiment may be included in at least one embodiment of the present application. The appearance of this phrase in various locations in the specification does not necessarily refer to the same embodiment, nor does it refer to independent or alternative embodiments that are mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0051] See Figure 1 , Figure 1The hardware structure diagram of a video encoding device provided in an embodiment of the present application is as follows: The video encoding device 100 includes at least one processor 101 , a communication circuit 102 , a memory 103 and at least one communication interface 104 .

[0052] In this embodiment, the processor 101 can be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present application.

[0053] The communication link 102 may include a path for transmitting information between the above components.

[0054] The communication interface 104 may be any transceiver-like device (eg, antenna, etc.) for communicating with other devices or communication networks, such as Ethernet, RAN, wireless local area networks (WLAN), etc.

[0055] The memory 103 may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.

[0056] In this embodiment, the memory 103 can exist independently and be connected to the processor 101 via the communication line 102. The memory 103 can also be integrated with the processor 101. The memory 103 provided in the embodiments of the present application can generally be non-volatile. Among them, the memory 103 is used to store computer-executable instructions for executing the solution of the present application, and the execution is controlled by the processor 101. The processor 101 is used to execute the computer-executable instructions stored in the memory 103, thereby implementing the methods provided in the following embodiments of the present application.

[0057] In an optional implementation, the computer-executable instructions may also be referred to as application code, which is not specifically limited in this application.

[0058] In an optional embodiment, the processor 101 may include one or more CPUs, such as Figure 1 CPU0 and CPU1 in.

[0059] In an optional embodiment, the video encoding apparatus 100 may include multiple processors, such as Figure 1 1 and 107. Each of these processors may be a single-CPU processor or a multi-CPU processor. A processor herein may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0060] In an optional embodiment, if the video encoding device 100 is a server, for example, it can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The video encoding device 100 may also include an output device 105 and an input device 106. The output device 105 communicates with the processor 101 and can display information in various ways. For example, the output device 105 can be a liquid crystal display (LCD), a light emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector. The input device 106 communicates with the processor 101 and can receive user input in various ways. For example, the input device 106 can be a mouse, a keyboard, a touch screen device, or a sensor device.

[0061] The video encoding apparatus 100 may be a general-purpose device or a dedicated device. The embodiment of the present application does not limit the type of the video encoding apparatus 100.

[0062] Secondly, it should be noted that the video encoding method provided in the embodiments of this application can be applied to scenarios where a first encoding process is converted to a second encoding process to save bitrate. Specifically, the bitrate consumption of the second encoding process is lower than that of the first encoding process, and the computing power consumption of the first encoding process is lower than that of the second encoding process.

[0063] For example, when the first encoding process is High Efficiency Video Coding (HEVC / H.265) encoding, the second encoding process can be the next generation of the Alliance of Open Media Video 1 (AV1) encoding. AV1 is a newer generation of video encoding. Compared with H.265 encoding, AV1 can bring more than 20% bit rate savings. Therefore, when H.265 encoding is converted to AV1 encoding, a large amount of bandwidth costs can be saved. However, while the AV1 video encoder brings bandwidth savings, because AV1 introduces more complex encoding tools, the encoding time of the AV1 encoder is also astonishing, which is 50 times that of the HEVC video encoder, which consumes huge computing power and increases computing power costs. The core of the huge computing power consumption of AV1 encoding is mainly due to the more complex coding block depth decision. In short, the computing power consumption of AV1 encoding is mainly concentrated in the confirmation of depth data. Therefore, through the video encoding method provided in the embodiments of the present application, while using AV1 encoding to replace H.265 encoding to save bit rate, it can smooth the demand for computing power costs and reduce the overall upgrade cost.

[0064] The following describes a video encoding method disclosed in this application using the conversion of H.265 encoding to AV1 encoding as an example. Therefore, the first encoding process will be represented by H.265 encoding, and the second encoding process will be represented by AV1 encoding.

[0065] See Figure 2 , Figure 2 A schematic diagram of a video encoding method provided in an embodiment of the present application. The video encoding method includes the following steps:

[0066] 201: Perform H.265 encoding on the video to be encoded to obtain first bitstream data.

[0067] In this embodiment, the first code stream data may be an H.265 code stream obtained after H.265 encoding.

[0068] 202: Extract depth information from the first bitstream data to obtain first depth map data.

[0069] In this embodiment, the first depth map data (Depth Map) can be established by extracting the depth information of each frame in the H.265 code stream. For example, the first depth map data for each frame of the video to be encoded can be established with a minimum granularity of 8. Specifically, a pre-established encoding relationship correspondence table can be queried based on the size of the coding unit in each frame to determine the depth value (Depth) of each coding unit in the first depth map data. At the same time, the area size of each coding unit in the first depth map data is determined based on the size of the coding unit and the minimum granularity.

[0070] In this embodiment, the encoding relationship correspondence table is shown in Table 1:

[0071] Table 1

[0072]

[0073]

[0074] In this embodiment, the area size of the coding unit in the first depth map data can be expressed by formula ①:

[0075]

[0076] Here, m represents the width of the region of the coding unit in the first depth map data, n represents the height of the region of the coding unit in the first depth map data, l represents the width of the coding unit, s represents the height of the coding unit, and b represents the minimum granularity.

[0077] Based on this, in this embodiment, the size of a coding unit in a certain frame in the H.265 code stream is 32x32, and the corresponding depth value can be found to be 2 according to the table. At the same time, since the minimum granularity is 8, the area size of the coding unit in the first depth map data is 4x4, and the depth map data corresponding to the coding unit is as follows Figure 3 shown.

[0078] Therefore, in this embodiment, by performing the above processing on each coding unit in each frame and splicing the obtained depth map data with the same positional relationship based on the positional relationship between each coding unit, the first depth map data can be obtained.

[0079] 203: Input the first depth map data into a depth prediction model to obtain second depth map data.

[0080] In this embodiment, for the depth value corresponding to the coordinate (x, y) in the second depth map data, the depth value in the area (x-a, ya)-(x+a, y+a) in the first depth map data can be input into the depth prediction model to obtain the depth value corresponding to the coordinate (x, y), where a is a natural number greater than or equal to 3. Then, all coordinates in the second depth map data are traversed to obtain the second depth map data.

[0081] Specifically, when a=3, for the depth value at the position (x, y) in the second depth map data, the depth data in the range of (x-3, y-3)-(x+3, y+3) in the first depth map data, that is, the depth data in the area of ​​7x7 centered on (x, y) is used as the feature of the input depth prediction model for prediction.

[0082] In this embodiment, when the coordinates (x, y) are the coordinates of the boundary area, the area (x-a, ya)-(x+a, y+a) may exceed the area of ​​the first depth map data. At this time, the boundary area of ​​the first depth map data can be extended and assigned before prediction to obtain third depth map data, and the first depth map data can be updated to the third depth map data to ensure that there will be no null value in the area (x-a, ya)-(x+a, y+a). Specifically, the first depth map data corresponding to the H.265 code stream will extend the Margin, and the Margin is extended and assigned by the edge Depth value, that is, the Depth value in the extended area is determined by the nearest known Depth value.

[0083] In this embodiment, the depth prediction model can be a classifier, such as an XGBoost classifier. For example, the depth prediction model can be obtained by pre-training an initial classification model using a set of training videos. The set of training videos can be historical experience data or artificial experience data, and this application does not impose any restrictions on this.

[0084] Specifically, the training process is as follows: First, obtain multiple training video sets. In this embodiment, any training video set includes at least one training video, the first reference depth map data of each training video in at least one training video, and the second reference depth map data of each training video, wherein the first reference depth map data is obtained after each training video is encoded by H.265, and the second reference depth map data is obtained after each training video is encoded by AV1. Then, call multiple prediction classifiers in the initial classification model to perform prediction processing on the first reference depth map data of each training video to obtain the prediction results of each training video. Finally, according to the prediction results of each training video and the second reference depth map data of each training video, the initial classification model is trained to obtain a depth prediction model.

[0085] In this embodiment, the specific process of calling multiple prediction classifiers in the initial classification model to perform prediction processing on the first reference depth map data of each training video to obtain the prediction results of each training video can be: traversing each prediction classifier in the initial classification model, based on the prediction results of each training video in the training video set corresponding to the last traversed prediction classifier obtained by processing the prediction classifier of the last traversal, determining the target training video set corresponding to the currently traversed target prediction classifier from the multiple training video sets. Then, calling the target prediction classifier, performing prediction processing on the first reference depth map data of each training video in the target training video set, and obtaining the prediction results of each training video in the target training video set.

[0086] Specifically, the specific process of traversing each prediction classifier in the initial classification model, and determining the target training video set corresponding to the target prediction classifier currently traversed from multiple training video sets based on the prediction results of each training video in the training video set corresponding to the prediction classifier of the previous traversal obtained by processing the prediction classifier of the previous traversal can be as follows: when the target prediction classifier is the first traversed prediction classifier, any one training video set is selected from the multiple training video sets as the target training video set corresponding to the target prediction classifier. When the target prediction classifier is not the first traversed prediction classifier, any one training video set is selected from the target training video set of the prediction classifier of the previous traversal whose prediction results are different from the second reference depth data corresponding to the target training video set, and any one training video set that has not participated in the training in the multiple training video sets as the target training video set corresponding to the target prediction classifier.

[0087] In this embodiment, the video data encoded in this way can also be added to the video training set of the depth prediction model as new historical experience data, and the depth prediction model is continuously trained and adjusted, thereby continuously deepening the accuracy of the depth prediction model.

[0088] 204: Determine encoding information for AV1 encoding according to the second depth map data.

[0089] In this embodiment, the encoding information may be the Block size information required when performing AV1 encoding. Specifically, the maximum encoding unit size of the video to be encoded can be first determined based on the resolution information of the video to be encoded, and then the corresponding depth data area is determined in the second depth map data based on the maximum encoding unit size. Then, the mode of each depth value in the depth data area is counted, and the mode of each depth value is arranged in order from large to small, and the depth values ​​corresponding to the first two modes of the arranged modes are extracted to obtain the first depth value and the second depth value. Finally, the pre-established encoding relationship correspondence table is queried based on the first depth value and the second depth value to determine the encoding information of the second encoding process.

[0090] For example, in AV1 encoding, the size information of the coding unit is shown in the rightmost column of Table 1. Therefore, the resolution information of the video to be encoded can be compared with the block sizes listed in the rightmost column of Table 1. The largest block size that is less than or equal to the resolution information of the video to be encoded can be selected as the maximum coding unit size.

[0091] Specifically, in AV1 encoding, the maximum block size is 128x128. Therefore, when the resolution of the video to be encoded is greater than 128x128, the maximum coding unit size of the video to be encoded is 128x128. Then, according to the determination method of formula (1) in step 202 above, when the minimum granularity is 8, the area size of the 128x128 coding unit in the second depth map data is 16x16. Then, based on the position information of the coding unit in the frame, the corresponding position in the second depth map data is determined for this 16x16 area, and a total of 256 depth values ​​in this area are obtained. Finally, take the first two values ​​of the 256 depth values ​​ranked by mode. For example, there are 100 0s, 90 1s, and 66 2s in the 256 values, and the first two values ​​ranked by mode are 0 and 1. Then, by looking up the table, we can determine the encoding information, that is, the required block size information is: 128x128, 64x64, 64x32, 32x64, 64x16, and 16x64. Therefore, when encoding the 128x128 coding unit, we will only try to use the block size corresponding to depth 0 and 1 (128x128, 64x64, 64x32, 32x64, 64x16, and 16x64) for encoding.

[0092] In this embodiment, when the encoding is smaller than a 128x128 coding unit, the mode is re-counted and the above process is repeated, but it is performed on the current coding unit size.

[0093] 205: Perform AV1 encoding on the video to be encoded according to the encoding information to obtain an encoded video.

[0094] In summary, in the video encoding method provided by the present invention, the first bitstream data of the video to be encoded is obtained through a first encoding process with low computing power consumption, and then depth information is extracted from the first bitstream data to obtain the first depth map data of the video to be encoded. The first depth map data is then input into a pre-trained depth prediction model to obtain second depth map data for performing a second encoding process with low bit rate consumption. Finally, the corresponding encoding information is obtained based on the second depth map data, and the second encoding process is performed on the video to be encoded to obtain the encoded video. Thus, by predicting the first depth map data through the depth prediction model, the first encoding process with low computing power consumption is used to replace the second encoding process with the high computing power consumption to perform the depth map data acquisition step, thereby reducing the computing power requirement after replacing the second encoding process. At the same time, using the second encoding process with low bit rate consumption to complete the subsequent video encoding steps can save a lot of bit rate compared to the first encoding process. This achieves the goal of reducing bandwidth expenses while smoothing the demand for computing power costs.

[0095] See Figure 4 , Figure 4 This is a block diagram of the functional modules of a video encoding device provided in the embodiment of the present application. Figure 4 As shown, the video encoding device 400 includes:

[0096] A first encoding module 401 is configured to perform a first encoding process on the video to be encoded to obtain first bitstream data, and extract depth information from the first bitstream data to obtain first depth map data;

[0097] A prediction module 402 is configured to input the first depth map data into a depth prediction model to obtain second depth map data;

[0098] The second encoding module 403 is used to determine encoding information for a second encoding process based on the second depth map data, wherein the bit rate consumption of the second encoding process is lower than that of the first encoding process, and perform the second encoding process on the to-be-encoded video according to the encoding information to obtain an encoded video.

[0099] In an embodiment of the present invention, in inputting the first depth map data into the depth prediction model to obtain the second depth map data, the prediction module 402 is specifically configured to:

[0100] For the depth value corresponding to the coordinate (x, y) in the second depth map data, the depth value in the area (x–a, ya)-(x+a, y+a) in the first depth map data is input into the depth prediction model to obtain the depth value corresponding to the coordinate (x, y), where a is a natural number greater than or equal to 3;

[0101] All coordinates in the second depth map data are traversed to obtain the second depth map data.

[0102] In an embodiment of the present invention, before inputting the first depth map data into the depth prediction model to obtain the second depth map data, the prediction module 402 is further configured to:

[0103] Extending and assigning values ​​to a boundary area of ​​the first depth map data to obtain third depth map data;

[0104] The first depth map data is updated to the third depth map data.

[0105] In an embodiment of the present invention, in determining the encoding information of the second encoding process according to the second depth map data, the second encoding module 403 is specifically configured to:

[0106] Determining a maximum coding unit size of the video to be encoded according to resolution information of the video to be encoded;

[0107] Determining a corresponding depth data region in the second depth map data according to the maximum coding unit size;

[0108] Count the modes of the depth values ​​in the depth data area;

[0109] Arrange the modes of the depth values ​​in descending order, and extract the depth values ​​corresponding to the first two modes of the arranged modes to obtain the first depth value and the second depth value;

[0110] A pre-established encoding relationship correspondence table is searched according to the first depth value and the second depth value to determine encoding information for the second encoding process.

[0111] In an embodiment of the present invention, the video encoding apparatus 400 further includes a training module 404, specifically configured to:

[0112] Obtaining multiple training video sets, wherein each training video set includes at least one training video, first reference depth map data of each training video in the at least one training video, and second reference depth map data of each training video, wherein the first reference depth map data is obtained after each training video is subjected to a first encoding process, and the second reference depth map data is obtained after each training video is subjected to a second encoding process;

[0113] Calling multiple prediction classifiers in the initial classification model to perform prediction processing on the first reference depth map data of each training video to obtain prediction results for each training video;

[0114] The initial classification model is trained based on the prediction results of each training video and the second reference depth map data of each training video to obtain a depth prediction model.

[0115] In an embodiment of the present invention, in calling multiple prediction classifiers in the initial classification model to perform prediction processing on the first reference depth map data of each training video to obtain the prediction results of each training video, the training module 404 is specifically used to:

[0116] Traversing each prediction classifier in the initial classification model, based on the prediction results of each training video in the training video set corresponding to the last traversed prediction classifier obtained by processing the last traversed prediction classifier, determining the target training video set corresponding to the currently traversed target prediction classifier from multiple training video sets;

[0117] The target prediction classifier is called to perform prediction processing on the first reference depth map data of each training video in the target training video set to obtain a prediction result of each training video in the target training video set.

[0118] In an embodiment of the present invention, in terms of traversing each prediction classifier in the initial classification model, determining a target training video set corresponding to a currently traversed target prediction classifier from multiple training video sets based on prediction results of each training video in the training video set corresponding to the last traversed prediction classifier obtained by processing the last traversed prediction classifier, the training module 404 is specifically configured to:

[0119] When the target prediction classifier is the first traversed prediction classifier, any one training video set is selected from multiple training video sets as the target training video set corresponding to the target prediction classifier;

[0120] When the target prediction classifier is not the first traversed prediction classifier, any one of the training videos whose prediction results in the target training video set of the last traversed prediction classifier are different from the second reference depth data corresponding to the target training video set, and the training video sets that did not participate in the training in multiple training video sets is selected as the target training video set corresponding to the target prediction classifier.

[0121] See Figure 5 , Figure 5 This is a schematic diagram of the structure of an electronic device provided in the embodiment of this application. Figure 5 As shown, the electronic device 500 includes a transceiver 501, a processor 502, and a memory 503. These are connected via a bus 504. The memory 503 is used to store computer programs and data, and can transmit the data stored in the memory 503 to the processor 502.

[0122] The processor 502 is configured to read the computer program in the memory 503 and perform the following operations:

[0123] Performing a first encoding process on the video to be encoded to obtain first code stream data;

[0124] Extracting depth information from the first bitstream data to obtain first depth map data;

[0125] Inputting the first depth map data into a depth prediction model to obtain second depth map data;

[0126] Determining encoding information for a second encoding process based on the second depth map data, wherein the bit rate consumption of the second encoding process is lower than that of the first encoding process, and the computing power consumption of the first encoding process is lower than that of the second encoding process;

[0127] A second encoding process is performed on the video to be encoded according to the encoding information to obtain an encoded video.

[0128] In an embodiment of the present invention, in terms of inputting the first depth map data into the depth prediction model to obtain the second depth map data, the processor 502 is specifically configured to perform the following operations:

[0129] For the depth value corresponding to the coordinate (x, y) in the second depth map data, the depth value in the area (x–a, ya)-(x+a, y+a) in the first depth map data is input into the depth prediction model to obtain the depth value corresponding to the coordinate (x, y), where a is a natural number greater than or equal to 3;

[0130] All coordinates in the second depth map data are traversed to obtain the second depth map data.

[0131] In an embodiment of the present invention, before inputting the first depth map data into the depth prediction model to obtain the second depth map data, the processor 502 is further configured to perform the following operations:

[0132] Extending and assigning values ​​to a boundary area of ​​the first depth map data to obtain third depth map data;

[0133] The first depth map data is updated to the third depth map data.

[0134] In an embodiment of the present invention, in determining the encoding information of the second encoding process according to the second depth map data, the processor 502 is specifically configured to perform the following operations:

[0135] Determining a maximum coding unit size of the video to be encoded according to resolution information of the video to be encoded;

[0136] Determining a corresponding depth data region in the second depth map data according to the maximum coding unit size;

[0137] Count the modes of the depth values ​​in the depth data area;

[0138] Arrange the modes of the depth values ​​in descending order, and extract the depth values ​​corresponding to the first two modes of the arranged modes to obtain the first depth value and the second depth value;

[0139] A pre-established encoding relationship correspondence table is searched according to the first depth value and the second depth value to determine encoding information for the second encoding process.

[0140] In this embodiment of the present invention, the processor 502 is further configured to perform the following operations:

[0141] Obtaining multiple training video sets, wherein each training video set includes at least one training video, first reference depth map data of each training video in the at least one training video, and second reference depth map data of each training video, wherein the first reference depth map data is obtained after each training video is subjected to a first encoding process, and the second reference depth map data is obtained after each training video is subjected to a second encoding process;

[0142] Calling multiple prediction classifiers in the initial classification model to perform prediction processing on the first reference depth map data of each training video to obtain prediction results for each training video;

[0143] The initial classification model is trained based on the prediction results of each training video and the second reference depth map data of each training video to obtain a depth prediction model.

[0144] In an embodiment of the present invention, in calling multiple prediction classifiers in the initial classification model to perform prediction processing on the first reference depth map data of each training video to obtain the prediction results of each training video, the processor 502 is specifically configured to perform the following operations:

[0145] Traversing each prediction classifier in the initial classification model, based on the prediction results of each training video in the training video set corresponding to the last traversed prediction classifier obtained by processing the last traversed prediction classifier, determining the target training video set corresponding to the currently traversed target prediction classifier from multiple training video sets;

[0146] The target prediction classifier is called to perform prediction processing on the first reference depth map data of each training video in the target training video set to obtain a prediction result of each training video in the target training video set.

[0147] In an embodiment of the present invention, in terms of traversing each prediction classifier in the initial classification model, determining a target training video set corresponding to a currently traversed target prediction classifier from multiple training video sets based on prediction results of each training video in the training video set corresponding to the last traversed prediction classifier obtained by processing the last traversed prediction classifier, the processor 502 is specifically configured to perform the following operations:

[0148] When the target prediction classifier is the first traversed prediction classifier, any one training video set is selected from multiple training video sets as the target training video set corresponding to the target prediction classifier;

[0149] When the target prediction classifier is not the first traversed prediction classifier, any one of the training videos whose prediction results in the target training video set of the last traversed prediction classifier are different from the second reference depth data corresponding to the target training video set, and the training video sets that did not participate in the training in multiple training video sets is selected as the target training video set corresponding to the target prediction classifier.

[0150] It should be understood that the video encoding devices in this application may include smartphones (such as Android phones, iOS phones, Windows Phone phones, etc.), tablet computers, PDAs, laptop computers, mobile Internet devices (MIDs), robots, or wearable devices. The above-mentioned video encoding devices are merely examples, not exhaustive, and include but are not limited to the above-mentioned video encoding devices. In practical applications, the above-mentioned video encoding devices may also include: smart vehicle terminals, computer equipment, etc.

[0151] Through the description of the above embodiments, those skilled in the art can clearly understand that the present invention can be implemented by combining software with a hardware platform. Based on this understanding, all or part of the contribution of the technical solution of the present invention to the background art can be embodied in the form of a software product. This computer software product can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments of the present invention or certain parts of the embodiments.

[0152] Therefore, embodiments of the present application further provide a computer-readable storage medium storing a computer program, wherein the computer program is executed by a processor to implement some or all of the steps of any of the video encoding methods described in the above method embodiments. For example, the storage medium may include a hard disk, a floppy disk, an optical disk, a magnetic tape, a magnetic disk, a USB flash drive, a flash memory, or the like.

[0153] An embodiment of the present application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute part or all of the steps of any one of the video encoding methods described in the above method embodiments.

[0154] It should be noted that for the aforementioned method implementations, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the implementations described in the specification are all optional implementations, and the actions and modules involved are not necessarily required for this application.

[0155] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0156] In the several embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0157] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of this embodiment.

[0158] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or in the form of software program modules.

[0159] If the integrated unit is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes a number of instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned memory includes various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0160] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable memory, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0161] The above is a detailed introduction to the implementation methods of the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above implementation methods is only used to help understand the method and core idea of ​​the present application. At the same time, for those skilled in the art, based on the ideas of the present application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A video encoding method, characterized in that: The method comprises: Performing a first encoding process on the video to be encoded to obtain first code stream data; Extracting depth information from the first bitstream data to obtain first depth information; Input the depth value in the area (x–a, ya)-(x+a, y+a) in the first depth information into the depth prediction model to obtain the depth value corresponding to the coordinate (x, y) in the second depth information, where a is a natural number greater than or equal to 3; Traverse all coordinates in the second depth information to obtain the second depth information; Determining a maximum coding unit size of the video to be encoded according to resolution information of the video to be encoded; determining, in the second depth information, a corresponding depth data region according to the maximum coding unit size; Counting the mode of each depth value in the depth data area; Arrange the modes of the depth values ​​in descending order, and extract the depth values ​​corresponding to the first two modes of the arranged modes to obtain a first depth value and a second depth value; determining encoding information for a second encoding process by querying a pre-established encoding relationship correspondence table according to the first depth value and the second depth value, the pre-established encoding relationship correspondence table including a correspondence between depth values ​​and sizes of coding units, wherein the bit rate consumption of the second encoding process is lower than that of the first encoding process, and the computing power consumption of the first encoding process is lower than that of the second encoding process; The second encoding process is performed on the video to be encoded according to the encoding information to obtain an encoded video.

2. The method according to claim 1, characterized in that Before inputting the depth value in the area (x−a, ya)-(x+a, y+a) in the first depth information into the depth prediction model to obtain the depth value corresponding to the coordinate (x, y) in the second depth information, the method further includes: Extending and assigning a value to a boundary area of ​​the first depth information to obtain third depth information; The first depth information is updated to the third depth information.

3. The method according to claim 1 or 2, characterized in that The method further comprises: Obtaining multiple training video sets, wherein each training video set includes at least one training video, first reference depth information of each training video in the at least one training video, and second reference depth information of each training video, wherein the first reference depth information is obtained after the first encoding process is performed on each training video, and the second reference depth information is obtained after the second encoding process is performed on each training video; Calling multiple prediction classifiers in the initial classification model to perform prediction processing on the first reference depth information of each training video to obtain prediction results for each training video; The initial classification model is trained according to the prediction results of the respective training videos and the second reference depth information of the respective training videos to obtain the depth prediction model.

4. The method according to claim 3, characterized in that The calling of multiple prediction classifiers in the initial classification model to perform prediction processing on the first reference depth information of each training video to obtain prediction results of each training video includes: Traversing each prediction classifier in the initial classification model, based on the prediction results of each training video in the training video set corresponding to the last traversed prediction classifier obtained by processing the last traversed prediction classifier, determining a target training video set corresponding to the currently traversed target prediction classifier from the multiple training video sets; The target prediction classifier is called to perform prediction processing on the first reference depth information of each training video in the target training video set to obtain a prediction result for each training video in the target training video set.

5. The method according to claim 4, characterized in that The traversing each prediction classifier in the initial classification model, determining a target training video set corresponding to the currently traversed target prediction classifier from the multiple training video sets based on prediction results of each training video in the training video set corresponding to the last traversed prediction classifier obtained by processing the last traversed prediction classifier, and comprising: When the target prediction classifier is the first traversed prediction classifier, selecting any one training video set from the multiple training video sets as the target training video set corresponding to the target prediction classifier; When the target prediction classifier is not the first traversed prediction classifier, any one of the training videos whose prediction results are different from the second reference depth data corresponding to the target training video set from the target training video set of the last traversed prediction classifier, and the training video sets that did not participate in the training in the multiple training video sets is selected as the target training video set corresponding to the target prediction classifier.

6. A video encoding device, characterized in that The device comprises: A first encoding module is configured to perform a first encoding process on the video to be encoded to obtain first bitstream data, and extract depth information from the first bitstream data to obtain first depth information; A prediction module, configured to input the depth value in the area (x–a, ya)-(x+a, y+a) in the first depth information into a depth prediction model to obtain a depth value corresponding to the coordinate (x, y) in the second depth information, where a is a natural number greater than or equal to 3; and traverse all coordinates in the second depth information to obtain the second depth information; A second encoding module is used to determine the maximum encoding unit size of the video to be encoded according to the resolution information of the video to be encoded; determine the corresponding depth data area in the second depth information according to the maximum encoding unit size; count the mode of each depth value in the depth data area; arrange the modes of the depth values ​​in descending order, and extract the depth values ​​corresponding to the first two modes of the arranged modes to obtain a first depth value and a second depth value; query a pre-established encoding relationship correspondence table according to the first depth value and the second depth value to determine the encoding information of the second encoding processing, the pre-established encoding relationship correspondence table includes the correspondence between the depth value and the size of the encoding unit, wherein the bit rate consumption of the second encoding processing is lower than that of the first encoding processing, and the computing power consumption of the first encoding processing is lower than that of the second encoding processing, and the second encoding processing is performed on the video to be encoded according to the encoding information to obtain the encoded video.

7. An electronic device, characterized in that: The method comprises a processor, a memory, a communication interface, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor, and the one or more programs include instructions for executing the steps in the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method according to any one of claims 1 to 5.

9. A computer program product, characterized in that The computer program product comprises a computer-readable storage medium storing a computer program, and the computer program is executed by a processor to implement the method according to any one of claims 1 to 5.

Images