Video processing method and device, related equipment, storage medium and computer program product

Abstract

The invention discloses a video processing method and device, related equipment, a storage medium and a computer program product. The method is applied to a first terminal, and comprises the following steps: determining a video frame containing an interested target based on original video data; the video frame containing the interested target comprises a key frame and a non-key frame; adjusting the coding parameter of the key frame to obtain the adjusted coding parameter; on the basis of the adjusted coding parameter, coding the key frame to obtain a first code stream; encoding key point information extracted from the non-key frame to obtain a second code stream; and sending the first code stream and the second code stream to a second terminal, so that the second terminal generates target video data.

Description

Technical Field

[0001] The present application relates to the field of communication technology, and in particular to a video processing method, apparatus, related equipment, storage medium and computer program product. Background Art

[0002] With the continuous development of science and technology, video calls are gradually becoming an important way of modern communication and collaboration, and the research on video reconstruction using neural networks has made significant progress in video calls. At present, in the process of using neural networks to learn two-dimensional (2D, 2-Dimension) or three-dimensional (3D, 3-Dimension) motion estimation and representation, since the static source image cannot provide enough appearance information for invisible areas or subtle expression changes, the complex motion in the driving video (such as large facial twists) causes the facial generation result to be blurred, resulting in serious artifacts and significantly reducing the quality of video image generation.

[0003] In order to solve this technical problem, a hybrid coding scheme with adjustable bit rate has been proposed in the related art. However, in extreme cases, such as when the channel bandwidth is extremely low, the noise signal is extremely large, or the face and other objects in the picture are greatly twisted, the video picture quality cannot meet the requirements of video calls, making it impossible for video calls to proceed normally in the above extreme cases. Summary of the invention

[0004] In order to solve the technical problems existing in the related technologies, the embodiments of the present application provide a video processing method, apparatus, related equipment, storage medium and computer program product.

[0005] To achieve the above purpose, the technical solution of the embodiment of the present application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides a video processing method, which is applied to a first terminal, and the method includes:

[0007] Based on the original video data, determining a video frame containing an object of interest; the video frame containing the object of interest includes a key frame and a non-key frame;

[0008] Adjusting the encoding parameters of the key frame to obtain adjusted encoding parameters;

[0009] Based on the adjusted encoding parameters, encoding the key frame to obtain a first bitstream;

[0010] Encoding the key point information extracted from the non-key frame to obtain a second bit stream;

[0011] The first code stream and the second code stream are sent to a second terminal, so that the second terminal generates target video data.

[0012] In a second aspect, an embodiment of the present application further provides another video processing method, which is applied to a second terminal, and the method includes:

[0013] Receiving a first code stream and a second code stream sent by a first terminal;

[0014] Generate target video data based on the first code stream and the second code stream;

[0015] The first bitstream is obtained by encoding the key frames in the video frames containing the target of interest determined based on the original video data by the first terminal based on the adjusted encoding parameters; the adjusted encoding parameters are obtained by adjusting the encoding parameters of the key frames; and the second bitstream is obtained by encoding the key point information extracted from the non-key frames of the video frames containing the target of interest by the first terminal.

[0016] In a third aspect, an embodiment of the present application provides a video processing device, applied to a first terminal, the device comprising:

[0017] A first determining unit is used to determine a video frame containing an object of interest based on the original video data; the video frame containing the object of interest includes a key frame and a non-key frame;

[0018] A first adjustment unit, configured to adjust the encoding parameters of the key frame to obtain adjusted encoding parameters;

[0019] A first encoding unit, configured to encode the key frame based on the adjusted encoding parameter to obtain a first bitstream;

[0020] A second encoding unit, used for encoding the key point information extracted from the non-key frame to obtain a second code stream;

[0021] The sending unit is used to send the first code stream and the second code stream to a second terminal, so that the second terminal generates target video data.

[0022] In a fourth aspect, an embodiment of the present application further provides another video processing device, applied to a second terminal, the device comprising:

[0023] A first receiving unit, configured to receive a first code stream and a second code stream sent by a first terminal;

[0024] A first generating unit, configured to generate target video data based on the first code stream and the second code stream;

[0025] The first bitstream is obtained by encoding the key frames in the video frames containing the target of interest determined based on the original video data by the first terminal based on the adjusted encoding parameters; the adjusted encoding parameters are obtained by adjusting the encoding parameters of the key frames; and the second bitstream is obtained by encoding the key point information extracted from the non-key frames of the video frames containing the target of interest by the first terminal.

[0026] In a fifth aspect, an embodiment of the present application provides a first terminal, including: a first processor and a first memory for storing a computer program that can be run on the first processor;

[0027] Among them, when the first processor is used to run the computer program, it executes the steps of the video processing method on the first terminal side described in the embodiment of the present application.

[0028] In a sixth aspect, an embodiment of the present application further provides a second terminal, comprising: a second processor and a second memory for storing a computer program that can be run on the second processor;

[0029] Among them, the second processor is used to execute the steps of the video processing method on the second terminal side described in the embodiment of the present application when running the computer program.

[0030] In the seventh aspect, an embodiment of the present application also provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the video processing method on the first terminal side described in the embodiment of the present application are implemented, or the steps of the video processing method on the second terminal side described in the embodiment of the present application are implemented.

[0031] In the eighth aspect, an embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the video processing method on the first terminal side described in the embodiment of the present application, or implements the steps of the video processing method on the second terminal side described in the embodiment of the present application.

[0032] The video processing method, apparatus, related equipment, storage medium and computer program product provided by the embodiment of the present application, the first terminal determines the video frame containing the target of interest based on the original video data; the video frame containing the target of interest includes a key frame and a non-key frame; the encoding parameters of the key frame are adjusted to obtain the adjusted encoding parameters; based on the adjusted encoding parameters, the key frame is encoded to obtain a first code stream; the key point information extracted from the non-key frame is encoded to obtain a second code stream; the first code stream and the second code stream are sent to the second terminal so that the second terminal generates the target video data. The technical solution of the embodiment of the present application is adopted, by segmenting the original video data into the target of interest, only the video frame containing the target of interest is subjected to video hybrid encoding, that is, for the video key frame, by dynamically adjusting the encoding parameters of the key frame, and encoding the key frame based on the adjusted encoding parameters, and encoding the key point information extracted from the non-key frame, so that the second terminal can generate high-quality target video data, thereby improving the quality of video calls under extreme conditions to meet the requirements of video calls, and thus ensuring that video calls can be carried out normally under extreme conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A schematic diagram of a hybrid coding method with adjustable bit rate in the related art;

[0034] Figure 2 The video processing method of the present invention is shown in FIG. Figure 1 ;

[0035] Figure 3 The video processing method of the present invention is shown in FIG. Figure 2 ;

[0036] Figure 4 The video processing method of the present invention is shown in FIG. Figure 3 ;

[0037] Figure 5 A schematic diagram of a video call transmission process according to an embodiment of the present application;

[0038] Figure 6 A schematic diagram of an encoding and decoding process of an object of interest according to an embodiment of the present application;

[0039] Figure 7 The video processing device of the present invention is shown in FIG. Figure 1 ;

[0040] Figure 8 The video processing device of the present invention is shown in FIG. Figure 2 ;

[0041] Fig. 9 A schematic diagram of the hardware structure of the first terminal according to an embodiment of the present application;

[0042] Fig.10 A schematic diagram of the hardware structure of the second terminal according to an embodiment of the present application;

[0043] Fig.11 A schematic diagram of the composition structure of a video processing system according to an embodiment of the present application. DETAILED DESCRIPTION

[0044] The present application will be further described in detail below with reference to the accompanying drawings and embodiments.

[0045] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.

[0046] With the continuous development of science and technology, video calls are gradually becoming an important way of modern communication and collaboration. However, in remote areas or areas with poor network conditions (i.e., low bandwidth conditions), communication networks may experience congestion, low coverage, power fluctuations, and data restrictions. As an efficient communication method for interaction with intelligent agents (such as smart terminals), semantic communication extracts and encodes the needs and information features of intelligent agents and transmits them in noisy channels, thereby achieving accurate transmission of intelligent agent semantics, communication efficiency, and user experience quality. Compared with traditional grammatical communication, semantic communication does not require accurate transmission of data or communication symbols, but is more concerned with the matching degree between the semantic information input by the sender and the semantic information restored by the receiver. Generally speaking, the bandwidth occupied by semantic information is extremely low, making it possible for semantic communication to make video calls of the same quality as traditional methods at ultra-low bandwidth.

[0047] In recent years, research on video reconstruction using neural networks has made significant progress in both video reconstruction quality and robustness. Existing work mainly focuses on learning more accurate 2D or 3D motion estimation and representation to improve video generation quality. More specifically, it is to learn 2D facial key points to simulate the optical flow between the source image and the driving frame image in the driving video. For example, the first-order motion model (FOMM) is used to learn densely represented facial geometry in a self-supervised manner to simulate complex facial expression movements to produce more refined facial generation results, and by learning the motion information of key points, the object in the source image can be animated according to the action of the driving video. Video reconstruction.

[0048] However, in practical applications, no matter how accurately the motion is estimated and represented, since the static source image cannot provide sufficient appearance information for invisible areas or subtle changes in expression, complex motion in the driving video (such as large twists of the face) will cause the face generation result to be blurred, resulting in serious artifacts, which will significantly reduce the quality of video image generation. In order to solve this technical problem, a hybrid coding scheme with adjustable bit rate is proposed in the related art.

[0049] Figure 1 Schematic diagram of a hybrid coding method with adjustable bit rate in the related art, such as Figure 1 As shown in the figure, the key frames in the source image are sampled according to a certain frequency, and these key frames are encoded using traditional methods such as VVC (Versatile Video Coding), and non-key frames are reconstructed using deep generation methods based on the transmitted key points. Combining the pixel-level recovery capability of traditional encoding and the detail generation capability of deep learning, such a hybrid encoding scheme can synthesize high-quality face videos at ultra-low bit rates. The hybrid encoding scheme uses traditional encoding methods for key frames, semantic encoding for non-key frames to obtain key point information, and the encoding information of key frames and non-key frames is transmitted in the form of bitstream. After the receiving end uses the key point information and the key frame image for decoding, the encoding bit rate can be greatly reduced, and the jitter and ghosting between the face image and the background on the screen can be reduced.

[0050] In actual applications, after using this hybrid coding method, the data size generated by traditional key frame coding is much larger than the data size of non-key frame key points. Therefore, the quality of video calls is strongly related to the size of key frame images. When the key frame quantization parameter (QP) is adjusted or other frame rate prediction methods are used to adjust the key frame image quality, the key frame image compression size also changes accordingly. Therefore, in extreme cases, such as when the channel bandwidth is extremely low, the noise signal is extremely large, or the face and other objects in the picture are twisted significantly, the video image quality still cannot meet the requirements of video calls, making video calls unable to proceed normally in the above extreme cases.

[0051] Based on this, an embodiment of the present application proposes a video processing method. In various embodiments of the present application, the original video data is segmented into targets of interest, and video mixed encoding is performed only on video frames containing targets of interest. That is, for video key frames, the encoding parameters of the key frames are dynamically adjusted, and the key frames are encoded based on the adjusted encoding parameters, and the key point information extracted from non-key frames is encoded. In this way, the second terminal can generate high-quality target video data, thereby improving the quality of video calls under extreme conditions to meet the requirements of video calls, and thereby ensuring that video calls can proceed normally under extreme conditions.

[0052] The embodiment of the present application provides a video processing method, which is applied to a first terminal. Figure 2 The video processing method of the present invention is shown in FIG. Figure 1 ;like Figure 2 As shown, the method includes:

[0053] Step 201: Based on the original video data, determine the video frame containing the object of interest.

[0054] In the embodiment of the present application, the video frames containing the target of interest include key frames and non-key frames, and the first terminal represents the sending end of the video call.

[0055] In actual application, the first terminal first collects original video data, and then performs object segmentation on the collected original video data to obtain a video frame containing an object of interest.

[0056] Based on this, in one embodiment, determining a video frame containing a target of interest based on the original video data includes: separating the collected original video data to obtain a foreground image of the original video data; and extracting a target from the foreground image of the original video data to obtain the video frame containing the target of interest.

[0057] Here, the original video data includes at least one video frame, that is, one or more video frames, each of which may include an object of interest or may not include an object of interest. In order to improve the accuracy of video key point extraction information, after collecting the original video data under extreme conditions such as low bandwidth conditions, the embodiment of the present application performs target segmentation on the original video data to obtain a video frame containing an object of interest, wherein the object of interest may include but is not limited to: facial images (i.e., human faces), human body images, animal images, etc.

[0058] Here, the original video data includes a foreground image and a background image (also referred to as a background image). Since the foreground image reflects the scenery that is closer to the camera, that is, it reflects the main scenery, usually, the target of interest to the user is included in the foreground image. Therefore, after the first terminal in the embodiment of the present application collects the original video data, it first separates the foreground image from the original video data, and then extracts the target from the foreground image to obtain a video frame containing the target of interest.

[0059] It should be noted that the first terminal can select a corresponding target segmentation algorithm based on its current computing power performance, and use the selected target segmentation algorithm to separate the foreground image and the background image from the collected original video data, wherein the foreground image can be understood as a significant area containing the target of interest.

[0060] Step 202: adjusting the encoding parameters of the key frame to obtain adjusted encoding parameters.

[0061] In actual application, in one embodiment, before adjusting the encoding parameters of the key frame to obtain the adjusted encoding parameters, the method further includes: determining the key frame.

[0062] Here, the first terminal determines whether the video frame currently containing the target of interest is a key frame by calling the key frame judgment module, so as to encode the key frame and the non-key frame separately. In actual application, the first terminal can determine whether the video frame currently containing the target of interest is a key frame according to the size of the change value of the target area of ​​interest.

[0063] Based on this, in one embodiment, determining the key frame includes: determining a change value of an object region of interest between two adjacent video frames; and when the change value of the object region of interest is greater than a fifth threshold, determining the current frame in the two adjacent video frames as the key frame.

[0064] Here, the relevant information of the target area of ​​interest includes the image size of the target of interest (i.e., the object of interest), the image position of the target of interest, and the foreground mask (alpha matte) of the target of interest. It should be noted that the foreground mask can be understood as a grayscale image, which is composed of the grayscale values ​​of each pixel, wherein the grayscale value of each pixel represents the degree to which each pixel in the original video data belongs to the foreground object.

[0065] In actual application, when the target region of interest of two adjacent video frames changes greatly, it is determined that the scene of the two video frames changes greatly, and when the change rate is greater than the set threshold, the current frame is set as a key frame. The target region of interest change value can be understood as the foreground mask change rate of the target of interest in two adjacent video frames, and the fifth threshold can be preset according to actual needs, for example, it can be set to 0.05.

[0066] For example, assuming that the current frame is k and the adjacent frame of the current frame (such as the previous frame of the current frame) is k-1, the foreground mask change rate △ of the object of interest in the two video frames k and k-1 can be calculated by the following formula (1):

[0067]

[0068] Among them, k represents the current frame; k-1 represents the adjacent frame of the current frame; M represents the width of the target area of ​​interest; N represents the height of the target area of ​​interest; α(k,i,j) represents the foreground mask value of the i and j pixels in the k-th frame; α(k-1,i,j) represents the foreground mask value of the i and j pixels in the k-1-th frame; i and j represent pixels, respectively.

[0069] After the first terminal calculates the foreground mask change rate △ of the target of interest in the two video frames k and k-1, when △ is greater than the fifth threshold (for example, 0.05), it can be determined that the scene change of the two video frames exceeds the change threshold, and the current frame k can be determined as a key frame at this time.

[0070] In actual application, the first terminal can dynamically adjust the encoding parameters of the key frame based on the retransmission rate of the key point information of the non-key frame, and encode the key frame based on the adjusted encoding parameters to adaptively adjust the video picture quality according to network changes. The encoding parameters of the key frame include at least one of the key frame quantization parameter (i.e., key frame QP) and the key frame interval (GOP, Group of Pictures). The key frame interval can be understood as the number of frames between two key frames.

[0071] Based on this, in one embodiment, the encoding parameters of the key frame are adjusted to obtain the adjusted encoding parameters, including: determining the key point information retransmission rate for the non-key frame; based on the key point information retransmission rate, adjusting the encoding parameters of the key frame to obtain the adjusted encoding parameters.

[0072] The process of adjusting the encoding parameters of the key frames is described in detail below.

[0073] In actual application, the first terminal uses lossless compression for the key point information, that is, the first terminal compresses the data of the key point information and sends it to the second terminal (that is, the receiving end of the video call). During this process, if the second terminal detects that a certain data packet is lost, the second terminal sends a non-acknowledgement signal (NACK, Non-Acknowledge) message to the first terminal, and then the first terminal finds the corresponding data packet in the sending buffer according to the sequence number in the NACK message, and resends it to the second terminal, and records the retransmission rate of the data in the current time period (that is, the retransmission rate of the key point information of non-key frames).

[0074] Based on this, in one embodiment, determining the retransmission rate of key point information for the non-key frame includes: determining the amount of information to be retransmitted by the first terminal and the amount of information output by the first terminal in the lossless coding transmission process of the key point information; determining the retransmission rate of the key point information based on the amount of information to be retransmitted by the first terminal and the amount of information output by the first terminal.

[0075] Here, for determining the key point information retransmission rate based on the amount of information to be retransmitted by the first terminal and the amount of information output by the first terminal, the key point information retransmission rate can be determined based on the ratio of the amount of information to be retransmitted by the first terminal to the amount of information output by the first terminal.

[0076] Exemplarily, assuming that the amount of information to be retransmitted by the first terminal (i.e., the number of messages retransmitted by the first terminal) is represented as retrans, and the amount of information output by the first terminal (i.e., the number of messages output by the first terminal) is represented as outsegs, the key point information retransmission rate can be calculated by the following formula (2):

[0077]

[0078] Wherein, RR represents the retransmission rate of key point information (ie, retransmission rate (RR, Retransfer Radio)).

[0079] In actual application, after determining the key point information retransmission rate, the first terminal may adjust the key frame encoding parameters based on at least one of the key frame quantization parameter and the key frame interval.

[0080] Based on this, in one embodiment, adjusting the encoding parameters of the key frame based on the key point information retransmission rate to obtain the adjusted encoding parameters includes at least one of the following:

[0081] Based on the key point information retransmission rate, the key frame quantization parameter is adjusted to obtain an adjusted key frame quantization parameter;

[0082] Based on the key point information retransmission rate and the key point information change rate for the non-key frame, the key frame interval is adjusted to obtain an adjusted key frame interval.

[0083] Here, for the adjustment process of the encoding parameters of the key frames, the first terminal can adjust the key frame quantization parameters based on the key point information retransmission rate to obtain the adjusted key frame quantization parameters; it can also adjust the key frame interval based on the key point information retransmission rate and the key point information change rate for non-key frames to obtain the adjusted key frame interval; it can also adjust the key frame quantization parameters and the key frame interval at the same time, that is, while adjusting the key frame quantization parameters based on the key point information retransmission rate to obtain the adjusted key frame quantization parameters, the key frame interval is adjusted based on the key point information retransmission rate and the key point information change rate for non-key frames to obtain the adjusted key frame interval.

[0084] It should be noted that the encoding parameters of the key frame may also include the key frame appearance time. Specifically, based on the key point information retransmission rate, the key frame appearance time is adjusted to obtain the adjusted key frame appearance time. In practical applications, since the adjustment of the key frame quantization parameter and the key frame interval has a greater impact on the quality of the video call, while the adjustment of the key frame appearance time has a smaller impact on the quality of the video call, the embodiment of the present application focuses on the description of the adjustment process of the key frame quantization parameter and the key frame interval, and the adjustment process of the key frame appearance time is not described in detail.

[0085] The following is a detailed description of the adjustment process of the key frame quantization parameters and the key frame interval.

[0086] In actual application, after determining the key point information retransmission rate, the first terminal can adjust the key frame quantization parameter based on the comparison result between the key point information retransmission rate and the first threshold value, wherein the first threshold value can be preset according to actual needs, for example, can be set to 0.04.

[0087] Based on this, in one embodiment, the key frame quantization parameter is adjusted based on the key point information retransmission rate to obtain the adjusted key frame quantization parameter, including: when the key point information retransmission rate is greater than a first threshold, the key frame quantization parameter is increased to obtain the increased key frame quantization parameter; when the key point information retransmission rate is less than the first threshold, the key frame quantization parameter is reduced, and the reduced key frame quantization parameter is not lower than a preset value of the key frame quantization parameter.

[0088] Here, whether to adjust the key frame quantization parameter can be determined based on the size of the currently set key frame interval. When the key point information retransmission rate is greater than the first threshold and the currently set key frame interval meets the set conditions, the key frame quantization parameter is adjusted, that is, the key frame quantization parameter is increased; when the key point information retransmission rate is less than the first threshold, the key frame quantization parameter is reduced, and the reduced key frame quantization parameter is not lower than the preset value of the key frame quantization parameter.

[0089] It should be noted that the currently set key frame interval meets the set conditions, which can be understood as the currently set key frame interval can no longer be adjusted, that is, if it continues to be adjusted, it will exceed the target multiple of the key frame interval preset value, such as exceeding 2 times the key frame interval preset value.

[0090] Exemplarily, assuming that the first threshold is 0.04, when the key point information retransmission rate is greater than 0.04 and the currently set key frame interval cannot be adjusted, that is, the next adjustment will exceed 2 times the preset value of the key frame interval, then the key frame quantization parameter is adjusted, that is, the value of the key frame quantization parameter is increased by 1 each time the adjustment is made; when the key point information retransmission rate is less than 0.04, the key frame quantization parameter is reduced, but the reduced key frame quantization parameter is not lower than the preset value of the key frame quantization parameter.

[0091] In actual application, before adjusting the key frame interval based on the key point information retransmission rate and the key point information change rate for the non-key frame to obtain the adjusted key frame interval, the method also includes: determining the key point information change rate for the non-key frame.

[0092] In one embodiment, determining the change rate of key point information for the non-key frame includes: extracting key point information from the non-key frame based on semantic information of the non-key frame; determining movement change information of the extracted key point information, and determining the change rate of the key point information based on the movement change information of the key point information.

[0093] Here, after extracting the key point information in the non-key frame, the first terminal detects the movement change of each key point information, and calculates the key point information change rate by the following formula (3):

[0094]

[0095] Among them, n represents the number of key point information; Js represents the Jacobian matrix, which is the movement change matrix corresponding to the key point; det(Js) represents the determinant of the Jacobian matrix; det represents the average value of the determinant of each key point, that is, the change rate of the key point information.

[0096] In actual application, after determining the key point information change rate for non-key frames, the first terminal may adjust the key frame interval based on the key point information retransmission rate and the key point information change rate for non-key frames.

[0097] Based on this, in one embodiment, the key frame interval is adjusted based on the key point information retransmission rate and the key point information change rate for the non-key frame to obtain an adjusted key frame interval, including: when the key point information retransmission rate is less than a second threshold, based on the comparison result of the key point information change rate and a third threshold, the key frame interval is adjusted to obtain an adjusted key frame interval; when the key point information retransmission rate is greater than a fourth threshold, the key frame interval is increased, and the increased key frame interval does not exceed a target multiple of a preset value of the key frame interval; when the key point information retransmission rate is greater than or equal to the second threshold and less than or equal to the fourth threshold, the key frame interval is reduced, and the reduced key frame interval is not less than the preset value of the key frame interval.

[0098] Here, the second threshold can be pre-set according to actual needs, for example, it can be set to 0.001. When the key point information retransmission rate is less than the second threshold, it is necessary to further judge the key point information change rate, so as to adjust the key frame interval based on the comparison result between the key point information change rate and the third threshold. Among them, the third threshold can be pre-set according to actual needs. When different key point detection algorithms are selected, the value of the Jacobian matrix will be different, and the corresponding third threshold will also be different.

[0099] Here, for adjusting the key frame interval based on the comparison result of the key point information change rate and the third threshold value to obtain the adjusted key frame interval, specifically, when the key point information change rate is greater than the third threshold value, the key frame interval is reduced, and the reduced key frame interval is not lower than the frame rate of the video call; when the key point information change rate is less than the third threshold value, the key frame interval is increased, and the increased key frame interval is the initial key frame interval.

[0100] It should be noted that when the key frame interval is reduced, the reduced key frame interval is half of the initial key frame interval, or the reduced key frame interval is not lower than the frame rate of the video call. For example, assuming that the initial key frame interval is 50, the reduced key frame interval is 25. Considering the large number of key frames in conventional encoding, the key frame interval needs to be no lower than the frame rate of the video call, that is, if the frame rate of the video call is set to 15, the reduced key frame interval is no lower than 15, that is, the minimum value of the reduced key frame interval is 15.

[0101] Here, the fourth threshold value can be preset according to actual needs, for example, it can be set to 0.02. When the key point information retransmission rate is greater than the fourth threshold value, the key frame interval is increased, and the increased key frame interval is 1.2 times the initial key frame interval to reduce the overall transmission bit rate, but the increased key frame interval does not exceed the target multiple of the preset value of the key frame interval, for example, the increased key frame interval does not exceed 2 times the preset value of the key frame interval.

[0102] Step 203: Encode the key frame based on the adjusted encoding parameters to obtain a first bitstream.

[0103] Step 204: Encode the key point information extracted from the non-key frame to obtain a second bit stream.

[0104] In actual application, in one embodiment, before encoding the key point information extracted from the non-key frame to obtain the second bitstream, the method further includes: extracting the key point information from the non-key frame. Specifically, the first terminal may first obtain the semantic information of the non-key frame, and then extract the key point information from the non-key frame based on the semantic information of the non-key frame.

[0105] Here, after extracting the key point information from the non-key frame, the first terminal does not need to adjust the original encoding parameters of the non-key frame, but encodes the key point information extracted from the non-key frame based on the original encoding parameters of the non-key frame to obtain a second code stream.

[0106] Step 205: Send the first code stream and the second code stream to a second terminal, so that the second terminal generates target video data.

[0107] In the embodiment of the present application, the second terminal is the opposite end of the first terminal, that is, the second terminal represents the receiving end of the video call.

[0108] Here, the first terminal may use an unreliable transmission method, such as the User Datagram Protocol (UDP), to send the first code stream to the second terminal. That is, in order to ensure the real-time nature of the key frame information, the data will not be retransmitted after loss. Among them, UDP is a connectionless transport layer protocol that provides a simple transaction-oriented unreliable information transmission service. In other words, it provides a method for applications to send encapsulated IP data packets without establishing a connection. The protocol does not provide a guarantee mechanism for data transmission. If the data packet is lost during the transmission from the first terminal to the second terminal, the protocol itself cannot make any detection or prompt. Therefore, the UDP protocol is usually called an unreliable transmission protocol. The first terminal may use a reliable transmission method to send the second code stream to the second terminal.

[0109] In practical applications, for key point information and video background image information (referred to as background information for short), retransmission technology can be used to ensure that the information is reachable.

[0110] Based on this, in one embodiment, the original video data includes video background image information; the method also includes: receiving non-confirmation signal information sent by the second terminal; based on the non-confirmation signal information, retransmitting the key point information and the video background image information to the second terminal.

[0111] Here, after receiving the non-confirmation signal information sent by the second terminal, the first terminal may retransmit the key point information and the video background image information to the second terminal in a reliable transmission manner.

[0112] Accordingly, the embodiment of the present application further provides another video processing method, which is applied to the second terminal. Figure 3 The video processing method of the present invention is shown in FIG. Figure 2 ;like Figure 3 As shown, the method includes:

[0113] Step 301: Receive a first code stream and a second code stream sent by a first terminal.

[0114] Step 302: Generate target video data based on the first code stream and the second code stream.

[0115] In an embodiment of the present application, the first bitstream is obtained by the first terminal encoding key frames in a video frame containing an object of interest determined based on original video data based on adjusted encoding parameters; the adjusted encoding parameters are obtained by adjusting encoding parameters of the key frames; and the second bitstream is obtained by the first terminal encoding key point information extracted from non-key frames of the video frame containing the object of interest.

[0116] It should be noted that the generation process of the first code stream and the second code stream can be understood by referring to the aforementioned video processing method on the first terminal side, and will not be described in detail here.

[0117] In actual application, the second terminal can reconstruct the target video data based on the decoding information of the first code stream and the decoding information of the second code stream to form a new video picture and display it.

[0118] Based on this, in one embodiment, after generating the target video data based on the first code stream and the second code stream, the method further includes: displaying the target video data.

[0119] In one embodiment, generating target video data based on the first code stream and the second code stream includes: decoding the first code stream to obtain key frame decoding information; decoding the second code stream to obtain key point decoding information; and generating the target video data based on the key frame decoding information and the key point decoding information.

[0120] In actual application, in one embodiment, the target video data is generated based on the key frame decoding information and the key point decoding information, including: reconstructing the non-key frame based on the key frame decoding information and the key point decoding information to obtain a reconstructed non-key frame; generating the target video data based on the reconstructed non-key frame and the key frame decoding information.

[0121] Here, after receiving the first code stream, the second terminal decodes the first code stream, i.e., the key frame, to obtain corresponding key frame decoding information. At the same time, after receiving the second code stream, the second terminal decodes the key point information of the second code stream, i.e., the non-key frame, to obtain key point decoding information; using the key frame decoding information and the key point decoding information of the non-key frame, the non-key frame is reconstructed to obtain a reconstructed non-key frame, and then a reconstructed video frame is generated based on the reconstructed non-key frame, and the reconstructed video frame is mixed with the background image to form a new video picture, i.e., the target video data.

[0122] Based on this, in one embodiment, before generating the target video data based on the reconstructed non-key frame and the key frame decoding information, the method also includes: receiving background image encoding information sent by the first terminal; decoding the background image encoding information to obtain video background image information.

[0123] In one embodiment, the generating of the target video data based on the reconstructed non-key frames and the key frame decoding information includes: generating a reconstructed video frame based on the reconstructed non-key frames and the key frame decoding information; generating the target video data based on the reconstructed video frame and the video background image information.

[0124] Here, the video background image can be selected by the first terminal, and when the video call is initially established, the first terminal encodes the video background image to obtain background image encoding information, and sends the background image encoding information to the second terminal via a reliable transmission method; after receiving the compressed (i.e., encoded) background image encoding information, the second terminal decompresses (i.e., decodes) the background image encoding information to obtain the video background image information.

[0125] It should be noted that the first terminal and the second terminal can also negotiate the background template and its corresponding identifier in advance. In this way, when the video call is initially established, the first terminal sends the identifier of the preset background template to the second terminal. After receiving the identifier of the background template sent by the first terminal, the second terminal extracts the corresponding background information from the background template library.

[0126] The embodiment of the present application also provides another video processing method, which is an interactive method between a first terminal and a second terminal. Figure 4 The video processing method of the present invention is shown in FIG. Figure 3 ;like Figure 4 As shown, the method includes:

[0127] Step 401: The first terminal determines a video frame containing an object of interest based on original video data.

[0128] In the embodiment of the present application, the video frames containing the object of interest include key frames and non-key frames.

[0129] Step 402: The first terminal adjusts the encoding parameters of the key frame to obtain adjusted encoding parameters, and encodes the key frame based on the adjusted encoding parameters to obtain a first bitstream.

[0130] Step 403: The first terminal encodes the key point information extracted from the non-key frame to obtain a second code stream.

[0131] Step 404: The first terminal sends the first code stream and the second code stream to the second terminal.

[0132] Step 405: The second terminal receives the first code stream and the second code stream sent by the first terminal.

[0133] Step 406: The second terminal generates target video data based on the first code stream and the second code stream.

[0134] It should be noted that the specific process of the first terminal and the second terminal completing the video processing has been described in detail above and will not be repeated here.

[0135] By adopting the technical solution of the embodiment of the present application, the original video data is segmented into targets of interest, and video mixed encoding is performed only on the video frames containing the targets of interest. That is, for video key frames, the encoding parameters of the key frames are dynamically adjusted, and the key frames are encoded based on the adjusted encoding parameters, and the key point information extracted from the non-key frames is encoded. In this way, the second terminal can generate high-quality target video data, thereby improving the quality of video calls under extreme conditions to meet the requirements of video calls, and further ensuring that video calls can proceed normally under extreme conditions.

[0136] The present application is described below in conjunction with application examples.

[0137] In view of the problem in the related art that in extreme cases, such as when the channel bandwidth is extremely low, the noise signal is extremely large, or the face and other targets in the picture are twisted significantly, the video picture quality cannot meet the requirements of the video call, making the video call unable to proceed normally in the above extreme cases, this application collects the call video (corresponding to the aforementioned original video data) in the above extreme cases, sets the target of interest, segments the target of interest in the original video picture, and only performs video encoding on the video frames containing the target of interest, including key frame encoding and key point extraction of non-key frames. Normally, the target of interest in a video call can be a human face, the entire human body, or an animal, etc. When the target of interest is relatively simple, such as extracting key points only for the face, the key point information is relatively simple, and the video picture reconstruction result is more realistic.

[0138] In this application, for key point information and background information, retransmission technology is used to ensure the accessibility of information; for key frame information, in order to ensure the real-time nature of the information, data is not retransmitted after loss; the sending end (corresponding to the aforementioned first terminal) dynamically adjusts the key frame quantization parameter (QP) size and key frame interval (GOP) according to the key point information retransmission rate request and key point information changes sent by the receiving end (corresponding to the aforementioned second terminal). In this way, a video with a higher resolution can be transmitted at a low bit rate under the existing terminal computing power performance, reducing the image ghosting when the target of interest moves and twists, and adaptively adjusting the video image quality according to network changes.

[0139] In order to improve the accuracy of video key point extraction information, the hybrid coding scheme adopted by related technologies, that is, the key frames adopt the traditional coding method, and the non-key frames adopt the semantic coding to obtain the key point information, such as Figure 5The schematic diagram of the video call transmission process shown in the figure shows that before the hybrid coding scheme, the collected video (corresponding to the aforementioned original video data) is first segmented (i.e., the target of interest is extracted) to achieve the target segmentation of the target of interest such as human face, human body or other animals. Subsequently, only the foreground image of the extracted target of interest is subjected to video hybrid coding, including key frame coding (unreliable transmission) and key point extraction of non-key frame (reliable transmission). For the video background, the video background can be selected at the sending end, and the background image can be encoded (i.e., background encoding) at the beginning of the video call establishment, and transmitted to the receiving end through reliable transmission. The receiving end decompresses the received compressed background image (i.e., background decoding). After the receiving end receives the reconstructed target of interest and reconstructs the target, it is mixed with the decoded background image to form a new video picture (corresponding to the aforementioned target video data) and displayed. In practical applications, the receiving end and the sending end may also negotiate the background template and its corresponding identifier in advance. The sending end sends the identification number of the background template set in advance at the beginning of the video call establishment. After the receiving end receives the identification number of the background template, it extracts the corresponding background from the background template library and mixes it with the target of interest to form a new video picture and display it.

[0140] Figure 6 This is a schematic diagram of the encoding and decoding process of the target of interest in the embodiment of the present application. Figure 6 According to the computing power performance of the current sending end, the object of interest extraction module selects a corresponding target segmentation algorithm such as a deep learning-based target segmentation algorithm (MODNet, Matting Objective Decomposition Network). The object of interest extraction module is used to predict the salient area in the captured video so as to separate the foreground and background images from the captured video. The object of interest extraction module outputs the region-related information of the object of interest, including the image size, image position and foreground mask of the object of interest.

[0141] When the sender determines that the frame image is a non-key frame, the non-key frame key point detection module is called, and the non-key frame key point detection module is used to perform key point detection on the object area of ​​interest, that is, a deep learning algorithm for posture key point detection, such as MoveNet or key point detection (KP-Detector, Key Points Detector), is used to predict the key points Hs and the corresponding Jacobian matrix Js of the object area of ​​interest. Each key point Hs consists of an (x, y) coordinate, and the size of the Jacobian matrix corresponding to the key point is 2×2. Lossless compression is used for Hs and Js. The sender compresses the key point information and sends it to the receiver. If a data frame is lost, the receiver sends NACK to the sender. The sender needs to retransmit the data frame and record the retransmission rate (RR) of the data in the current time period. RR can be obtained by the above formula (2).

[0142] The key frame judgment module receives the region-related information of the object of interest transmitted by the object of interest extraction module, including the image size, image position and foreground mask of the object of interest, and receives the retransmission rate (corresponding to the aforementioned retransmission rate of key point information for non-key frames) and the detection key point change rate (corresponding to the aforementioned change rate of key point information for non-key frames) transmitted by the non-key frame key point detection module. According to the large change in the region of interest, the scene change is determined to be large. When the change in the target region of interest exceeds a certain threshold (corresponding to the aforementioned fifth threshold), the current frame is set as a key frame. In accordance with the principle of reducing the GOP when the non-key frame key point change rate is large and increasing the GOP when the retransmission rate increases, the GOP is dynamically adjusted to determine whether the current frame is a key frame, and output the video frame to the key frame encoding and non-key frame key point detection modules respectively.

[0143] When RR is less than 0.001, the key point change rate is further determined. When the key point change rate is greater than a certain threshold value Ts, the GOP is reduced and the reduced GOP is set to half of the initial GOP. If the initial GOP is 50, the GOP is adjusted to 25. Considering the large number of key frames in conventional encoding, the reduced GOP needs to be no less than the video call frame rate, that is, if the video call frame rate is set to 15, the minimum value of the reduced GOP is 15. When the key point change rate is lower than Ts, the GOP is readjusted to the initial preset value. If RR is greater than 0.02, the GOP is increased and the increased GOP is set to 1.2 times the initial GOP (GOPnew = GOPold*1.2) to reduce the overall transmission bit rate, but the increased GOP size does not exceed 2 times the preset GOP value. When RR is less than 0.02 and greater than 0.001, the GOP is reduced again and adjusted to GOPnew = GOPold / 1.2, but the adjusted GOP is not lower than the initial preset value.

[0144] The key frame encoding module performs key frame encoding on the target area of ​​interest. When receiving the RR transmitted by the non-key frame key point detection module, it determines whether to adjust the QP according to the currently set GOP size. When the RR is greater than 0.04 and the GOP cannot be adjusted, that is, the next adjustment will exceed 2 times the preset GOP value, the QP is adjusted, and the QP value is increased by 1 each time; when the RR is lower than 0.04, the QP value is reduced, but the reduced QP value is not lower than the preset QP value.

[0145] It should be noted that after the key frame encoding module encodes the current key frame, it is transmitted to the receiving end using an unreliable transmission method, such as the UDP protocol. After receiving the key frame, the receiving end decodes the key frame; at the same time, the non-key frame is reconstructed using the key frame decoding information and the non-key frame key point information, and then the receiving end mixes the reconstructed video frame with the background image to form a new video picture and displays it.

[0146] Compared with the methods of related technologies, this application has the following beneficial effects:

[0147] When the channel bandwidth is extremely low, the noise signal is extremely large, or the face and other objects in the picture are greatly twisted, the picture quality of the related art solutions cannot meet the requirements of video calls. This application can improve the video call quality in the above extreme situations by dynamically adjusting the GOP and QP sizes; at the same time, when the channel bandwidth is good, this application can increase key frames and adjust QP, and the picture quality of the other end is greatly improved compared with the existing solutions.

[0148] In order to implement the video processing method on the first terminal side of the embodiment of the present application, the embodiment of the present application further provides a video processing device, which is arranged on the first terminal. Figure 7 The video processing device of the present invention is shown in FIG. Figure 1 ,like Figure 7 As shown, the device comprises:

[0149] A first determining unit 71 is used to determine a video frame containing an object of interest based on the original video data; the video frame containing the object of interest includes a key frame and a non-key frame;

[0150] A first adjustment unit 72, configured to adjust the encoding parameters of the key frame to obtain adjusted encoding parameters;

[0151] A first encoding unit 73, configured to encode the key frame based on the adjusted encoding parameter to obtain a first bit stream;

[0152] A second encoding unit 74, configured to encode the key point information extracted from the non-key frame to obtain a second bit stream;

[0153] The sending unit 75 is configured to send the first code stream and the second code stream to a second terminal, so that the second terminal generates target video data.

[0154] In one embodiment, the first determination unit 71 is specifically used to: separate the collected original video data to obtain a foreground image of the original video data; and extract a target from the foreground image of the original video data to obtain a video frame containing the target of interest.

[0155] In one embodiment, the first adjustment unit 72 includes:

[0156] A second determining unit, configured to determine a key point information retransmission rate for the non-key frame;

[0157] The second adjustment unit is used to adjust the encoding parameters of the key frame based on the key point information retransmission rate to obtain the adjusted encoding parameters.

[0158] In one embodiment, the second determination unit is specifically used to: determine the amount of information to be retransmitted by the first terminal and the amount of information output by the first terminal during the lossless coding transmission process of the key point information; and determine the retransmission rate of the key point information based on the amount of information to be retransmitted by the first terminal and the amount of information output by the first terminal.

[0159] In one embodiment, the encoding parameters of the key frame include at least one of a key frame quantization parameter and a key frame interval;

[0160] The second adjustment unit is specifically used for at least one of the following:

[0161] Based on the key point information retransmission rate, the key frame quantization parameter is adjusted to obtain an adjusted key frame quantization parameter;

[0162] Based on the key point information retransmission rate and the key point information change rate for the non-key frame, the key frame interval is adjusted to obtain an adjusted key frame interval.

[0163] In one embodiment, the device further comprises:

[0164] A third determining unit, configured to determine the key point information change rate for the non-key frame before the second adjusting unit adjusts the key frame interval based on the key point information retransmission rate and the key point information change rate for the non-key frame to obtain an adjusted key frame interval;

[0165] The third determination unit is specifically used to: extract key point information from the non-key frame based on the semantic information of the non-key frame; determine the movement change information of the extracted key point information, and determine the change rate of the key point information based on the movement change information of the key point information.

[0166] In one embodiment, the second adjustment unit is further specifically used to: when the key point information retransmission rate is greater than a first threshold, increase the key frame quantization parameter to obtain an increased key frame quantization parameter; when the key point information retransmission rate is less than the first threshold, reduce the key frame quantization parameter, and the reduced key frame quantization parameter is not lower than a preset value of the key frame quantization parameter.

[0167] In one embodiment, the second adjustment unit is further specifically used to: when the key point information retransmission rate is less than the second threshold, based on the comparison result of the key point information change rate and the third threshold, adjust the key frame interval to obtain an adjusted key frame interval; when the key point information retransmission rate is greater than the fourth threshold, increase the key frame interval, and the increased key frame interval does not exceed the target multiple of the key frame interval preset value; when the key point information retransmission rate is greater than or equal to the second threshold and less than or equal to the fourth threshold, reduce the key frame interval, and the reduced key frame interval is not less than the key frame interval preset value.

[0168] In one embodiment, the device further comprises:

[0169] a fourth determining unit, configured to determine the key frame before the first adjusting unit 72 adjusts the encoding parameters of the key frame to obtain the adjusted encoding parameters;

[0170] The fourth determination unit is specifically used to: determine a change value of the target region of interest between two adjacent video frames; and when the change value of the target region of interest is greater than a fifth threshold, determine a current frame in the two adjacent video frames as the key frame.

[0171] In one embodiment, the original video data includes video background image information; the device further includes:

[0172] A second receiving unit, configured to receive the non-confirmation signal information sent by the second terminal;

[0173] A retransmission unit is used to retransmit the key point information and the video background image information to the second terminal based on the unconfirmed signal information.

[0174] In actual application, the first determination unit 71, the first adjustment unit 72, the first encoding unit 73 and the second encoding unit 74 can be implemented by a processor in the video processing device; the sending unit 75 can be implemented by a communication interface in the video processing device.

[0175] It should be noted that: the video processing device provided in the above embodiment only uses the division of the above program modules as an example when performing video processing. In actual applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device is divided into different program modules to complete all or part of the processing described above. In addition, the video processing device provided in the above embodiment and the video processing method embodiment on the first terminal side belong to the same concept. The specific implementation process is detailed in the above-mentioned video processing method embodiment on the first terminal side, which will not be repeated here.

[0176] In order to implement the video processing method on the second terminal side of the embodiment of the present application, the embodiment of the present application also provides another video processing device, which is arranged on the second terminal. Figure 8 The video processing device of the present invention is shown in FIG. Figure 2 ,like Figure 8 As shown, the device comprises:

[0177] The first receiving unit 81 is used to receive a first code stream and a second code stream sent by a first terminal;

[0178] A first generating unit 82, configured to generate target video data based on the first code stream and the second code stream;

[0179] The first bitstream is obtained by encoding the key frames in the video frames containing the target of interest determined based on the original video data by the first terminal based on the adjusted encoding parameters; the adjusted encoding parameters are obtained by adjusting the encoding parameters of the key frames; and the second bitstream is obtained by encoding the key point information extracted from the non-key frames of the video frames containing the target of interest by the first terminal.

[0180] In one embodiment, the first generating unit 82 includes:

[0181] A first decoding unit, used for decoding the first bit stream to obtain key frame decoding information;

[0182] A second decoding unit, used for decoding the second code stream to obtain key point decoding information;

[0183] The second generating unit is used to generate the target video data based on the key frame decoding information and the key point decoding information.

[0184] In one embodiment, the second generating unit is specifically used to: reconstruct the non-key frame based on the key frame decoding information and the key point decoding information to obtain a reconstructed non-key frame; and generate the target video data based on the reconstructed non-key frame and the key frame decoding information.

[0185] In one embodiment, the device further comprises:

[0186] A third receiving unit, configured to receive background image encoding information sent by the first terminal before the second generating unit generates the target video data based on the reconstructed non-key frame and the key frame decoding information;

[0187] The third decoding unit is used to decode the background image encoding information to obtain video background image information.

[0188] In one embodiment, the second generating unit is further specifically used to: generate a reconstructed video frame based on the reconstructed non-key frame and the key frame decoding information; generate the target video data based on the reconstructed video frame and the video background image information.

[0189] In actual application, the first receiving unit 81 may be implemented by a communication interface in the video processing device, and the first generating unit 82 may be implemented by a processor in the video processing device.

[0190] It should be noted that: the video processing device provided in the above embodiment only uses the division of the above program modules as an example when performing video processing. In actual applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device is divided into different program modules to complete all or part of the processing described above. In addition, the video processing device provided in the above embodiment and the video processing method embodiment on the second terminal side belong to the same concept. The specific implementation process is detailed in the above-mentioned video processing method embodiment on the second terminal side, which will not be repeated here.

[0191] Based on the hardware implementation of the above program module, and in order to implement the video processing method on the first terminal side of the embodiment of the present application, the embodiment of the present application also provides a first terminal, Fig. 9 FIG. 1 is a schematic diagram of the hardware structure of the first terminal according to an embodiment of the present application. Fig. 9 As shown, the first terminal 90 includes:

[0192] The first communication interface 91 is capable of exchanging information with the second terminal;

[0193] The first processor 92 is connected to the first communication interface 91 to implement information interaction with the second terminal, and is used to execute the video processing method provided by one or more technical solutions on the first terminal side when running a computer program, and the computer program is stored in the first memory 93.

[0194] It should be noted that the specific processing process of the first communication interface 91 and the first processor 92 can be understood by referring to the above-mentioned video processing method on the first terminal side.

[0195] Of course, in actual application, the various components in the first terminal 90 are coupled together through the first bus system 94. It can be understood that the first bus system 94 is used to realize the connection and communication between these components. In addition to the data bus, the first bus system 94 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Fig. 9 In the figure, various buses are labeled as a first bus system 94.

[0196] The first memory 93 in the embodiment of the present application is used to store various types of data to support the operation of the first terminal 90. Examples of such data include: any computer program used to operate on the first terminal 90.

[0197] The video processing method on the first terminal side disclosed in the above embodiment of the present application can be applied to the first processor 92, or implemented by the first processor 92. The first processor 92 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the video processing method on the first terminal side can be completed by the hardware integrated logic circuit or software instructions in the first processor 92. The above-mentioned first processor 92 may be a general processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The first processor 92 can implement or execute the video processing methods, steps and logic block diagrams on the first terminal side disclosed in the embodiment of the present application. The general processor may be a microprocessor or any conventional processor, etc. In combination with the steps of the video processing method on the first terminal side disclosed in the embodiment of the present application, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in the first memory 93. The first processor 92 reads the information in the first memory 93 and completes the steps of the video processing method on the first terminal side in combination with its hardware.

[0198] In an exemplary embodiment, the first terminal 90 can be implemented by one or more application-specific integrated circuits (ASIC), DSP, programmable logic device (PLD), complex programmable logic device (CPLD), field-programmable gate array (FPGA), general-purpose processor, controller, microcontroller (MCU), microprocessor, or other electronic components to execute the aforementioned video processing method on the first terminal side.

[0199] Based on the hardware implementation of the above program module, and in order to implement the video processing method on the second terminal side of the embodiment of the present application, the embodiment of the present application also provides a second terminal, Fig.10 FIG. 1 is a schematic diagram of the hardware structure of the second terminal according to an embodiment of the present application. Fig.10 As shown, the second terminal 100 includes:

[0200] The second communication interface 101 is capable of exchanging information with the first terminal;

[0201] The second processor 102 is connected to the second communication interface 101 to implement information interaction with the first terminal, and is used to execute the video processing method provided by one or more technical solutions on the second terminal side when running a computer program, and the computer program is stored in the second memory 103.

[0202] It should be noted that the specific processing process of the second communication interface 101 and the second processor 102 can be understood by referring to the video processing method on the second terminal side.

[0203] Of course, in actual application, the various components in the second terminal 100 are coupled together through the second bus system 104. It can be understood that the second bus system 104 is used to realize the connection and communication between these components. In addition to the data bus, the second bus system 104 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Fig.10 In the figure, various buses are labeled as a second bus system 104 .

[0204] The second memory 103 in the embodiment of the present application is used to store various types of data to support the operation of the second terminal 100. Examples of such data include: any computer program used to operate on the second terminal 100.

[0205] The video processing method on the second terminal side disclosed in the above embodiment of the present application can be applied to the second processor 102, or implemented by the second processor 102. The second processor 102 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the video processing method on the second terminal side can be completed by the hardware integrated logic circuit or software instructions in the second processor 102. The above second processor 102 may be a general processor, DSP, or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The second processor 102 can implement or execute the video processing methods, steps and logic block diagrams on the second terminal side disclosed in the embodiment of the present application. The general processor may be a microprocessor or any conventional processor, etc. In combination with the steps of the video processing method on the second terminal side disclosed in the embodiment of the present application, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in the second memory 103. The second processor 102 reads the information in the second memory 103 and completes the steps of the video processing method on the second terminal side in combination with its hardware.

[0206] In an exemplary embodiment, the second terminal 100 may be implemented by one or more ASICs, DSPs, PLDs, CPLDs, FPGAs, general-purpose processors, controllers, MCUs, microprocessors, or other electronic components to execute the aforementioned video processing method on the second terminal side.

[0207] It can be understood that the memory (including the first memory 93 and the second memory 103) of the embodiment of the present application can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a ferromagnetic random access memory, a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM, SyncLink Dynamic Random Access Memory), and direct RAM bus random access memory (DRRAM, Direct Rambus Random Access Memory).The memories described in the embodiments of the present application are intended to include, but are not limited to, these and any other suitable types of memories.

[0208] In order to implement the video processing method of the embodiment of the present application, the embodiment of the present application also provides a video processing system. Fig.11 FIG. 1 is a schematic diagram of the composition structure of the video processing system according to an embodiment of the present application. Fig.11 As shown, the system includes: a first terminal 90 and a second terminal 100; wherein,

[0209] The first terminal 90 is used to determine a video frame containing an object of interest based on the original video data; the video frame containing the object of interest includes a key frame and a non-key frame; adjust the encoding parameters of the key frame to obtain the adjusted encoding parameters; encode the key frame based on the adjusted encoding parameters to obtain a first code stream; encode the key point information extracted from the non-key frame to obtain a second code stream; send the first code stream and the second code stream to the second terminal 100;

[0210] The second terminal 100 is used to receive the first code stream and the second code stream sent by the first terminal 90; and generate target video data based on the first code stream and the second code stream.

[0211] It should be noted that the specific process of the first terminal 90 and the second terminal 100 completing the video processing has been described in detail above and will not be repeated here.

[0212] In an exemplary embodiment, the embodiment of the present application further provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, for example, including a first memory 93 storing a computer program, the computer program can be executed by a first processor 92 in a first terminal 90 to complete the steps of the video processing method on the first terminal side, or including a second memory 103 storing a computer program, the computer program can be executed by a second processor 102 in a second terminal 100 to complete the steps of the video processing method on the second terminal side. The computer-readable storage medium can be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disk, or CD-ROM.

[0213] In an exemplary embodiment, the embodiment of the present application also provides a computer program product, including a computer program, which can be executed by a first processor 92 in a first terminal 90 to complete the steps of the video processing method on the first terminal side, or the computer program can be executed by a second processor 102 in a second terminal 100 to complete the steps of the video processing method on the second terminal side.

[0214] It should be noted that: "first", "second", "third", etc. are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0215] In addition, the technical solutions described in the embodiments of the present application can be combined arbitrarily without conflict.

[0216] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A video processing method, characterized in that: Applied to a first terminal, the method includes: Based on the original video data, determining a video frame containing an object of interest; the video frame containing the object of interest includes a key frame and a non-key frame; Adjusting the encoding parameters of the key frame to obtain adjusted encoding parameters; Based on the adjusted encoding parameters, encoding the key frame to obtain a first bitstream; Encoding the key point information extracted from the non-key frame to obtain a second bit stream; The first code stream and the second code stream are sent to a second terminal, so that the second terminal generates target video data.

2. The method according to claim 1, characterized in that The step of determining a video frame containing an object of interest based on the original video data comprises: Separating the collected original video data to obtain a foreground image of the original video data; The target is extracted from the foreground image of the original video data to obtain the video frame containing the target of interest.

3. The method according to claim 1, characterized in that The step of adjusting the encoding parameters of the key frame to obtain the adjusted encoding parameters includes: Determining a key point information retransmission rate for the non-key frame; Based on the key point information retransmission rate, the encoding parameters of the key frame are adjusted to obtain the adjusted encoding parameters.

4. The method according to claim 3, characterized in that The determining of the key point information retransmission rate for the non-key frame includes: Determine, for the lossless coding transmission process of the key point information, the amount of information to be retransmitted by the first terminal and the amount of information output by the first terminal; The key point information retransmission rate is determined based on the amount of information to be retransmitted by the first terminal and the amount of information output by the first terminal.

5. The method according to claim 3, characterized in that: The encoding parameters of the key frame include at least one of a key frame quantization parameter and a key frame interval; The step of adjusting the encoding parameters of the key frame based on the key point information retransmission rate to obtain the adjusted encoding parameters includes at least one of the following: Based on the key point information retransmission rate, the key frame quantization parameter is adjusted to obtain an adjusted key frame quantization parameter; Based on the key point information retransmission rate and the key point information change rate for the non-key frame, the key frame interval is adjusted to obtain an adjusted key frame interval.

6. The method according to claim 5, characterized in that Before adjusting the key frame interval based on the key point information retransmission rate and the key point information change rate for the non-key frame to obtain the adjusted key frame interval, the method further includes: Determining a rate of change of key point information for the non-key frame; The determining of the change rate of the key point information of the non-key frame includes: Extracting key point information from the non-key frame based on semantic information of the non-key frame; Determine the movement change information of the extracted key point information, and determine the change rate of the key point information based on the movement change information of the key point information.

7. The method according to claim 5, characterized in that The step of adjusting the key frame quantization parameter based on the key point information retransmission rate to obtain the adjusted key frame quantization parameter includes: When the key point information retransmission rate is greater than a first threshold, increasing the key frame quantization parameter to obtain an increased key frame quantization parameter; When the key point information retransmission rate is less than the first threshold, the key frame quantization parameter is reduced, and the reduced key frame quantization parameter is not less than a preset key frame quantization parameter value.

8. The method according to claim 5, characterized in that The step of adjusting the key frame interval based on the key point information retransmission rate and the key point information change rate for the non-key frame to obtain an adjusted key frame interval includes: When the key point information retransmission rate is less than the second threshold, based on the comparison result of the key point information change rate and the third threshold, adjusting the key frame interval to obtain an adjusted key frame interval; When the key point information retransmission rate is greater than a fourth threshold, the key frame interval is increased, and the increased key frame interval does not exceed a target multiple of a preset value of the key frame interval; When the key point information retransmission rate is greater than or equal to the second threshold and less than or equal to the fourth threshold, the key frame interval is reduced, and the reduced key frame interval is not less than the key frame interval preset value.

9. The method according to claim 1, characterized in that: Before adjusting the encoding parameters of the key frame to obtain the adjusted encoding parameters, the method further includes: Determining the key frame; The determining of the key frame comprises: Determine the change value of the target region of interest between two adjacent video frames; When the change value of the target region of interest is greater than a fifth threshold, a current frame of the two adjacent video frames is determined as the key frame.

10. The method according to claim 1, characterized in that The original video data includes video background image information; the method further includes: receiving a non-confirmation signal information sent by the second terminal; Based on the unconfirmed signal information, the key point information and the video background image information are retransmitted to the second terminal.

11. A video processing method, characterized in that: Applied to the second terminal, the method includes: Receiving a first code stream and a second code stream sent by a first terminal; Generate target video data based on the first code stream and the second code stream; The first bitstream is obtained by encoding the key frames in the video frames containing the target of interest determined based on the original video data by the first terminal based on the adjusted encoding parameters; the adjusted encoding parameters are obtained by adjusting the encoding parameters of the key frames; and the second bitstream is obtained by encoding the key point information extracted from the non-key frames of the video frames containing the target of interest by the first terminal.

12. The method according to claim 11, characterized in that The generating target video data based on the first code stream and the second code stream includes: Decoding the first code stream to obtain key frame decoding information; Decoding the second code stream to obtain key point decoding information; The target video data is generated based on the key frame decoding information and the key point decoding information.

13. The method according to claim 12, characterized in that The generating the target video data based on the key frame decoding information and the key point decoding information comprises: Reconstructing the non-key frame based on the key frame decoding information and the key point decoding information to obtain a reconstructed non-key frame; The target video data is generated based on the reconstructed non-key frame and the key frame decoding information.

14. The method according to claim 13, characterized in that Before generating the target video data based on the reconstructed non-key frame and the key frame decoding information, the method further includes: Receiving background image encoding information sent by the first terminal; Decoding the background image encoding information to obtain video background image information; The generating the target video data based on the reconstructed non-key frame and the key frame decoding information comprises: Generate a reconstructed video frame based on the reconstructed non-key frame and the key frame decoding information; The target video data is generated based on the reconstructed video frame and the video background image information.

15. A video processing device, characterized in that: Applied to a first terminal, the device includes: A first determining unit is used to determine a video frame containing an object of interest based on the original video data; the video frame containing the object of interest includes a key frame and a non-key frame; A first adjustment unit, configured to adjust the encoding parameters of the key frame to obtain adjusted encoding parameters; A first encoding unit, configured to encode the key frame based on the adjusted encoding parameter to obtain a first bitstream; A second encoding unit, used for encoding the key point information extracted from the non-key frame to obtain a second code stream; The sending unit is used to send the first code stream and the second code stream to a second terminal, so that the second terminal generates target video data.

16. A video processing device, characterized in that: Applied to a second terminal, the device includes: A first receiving unit, configured to receive a first code stream and a second code stream sent by a first terminal; A first generating unit, configured to generate target video data based on the first code stream and the second code stream; The first bitstream is obtained by encoding the key frames in the video frames containing the target of interest determined based on the original video data by the first terminal based on the adjusted encoding parameters; the adjusted encoding parameters are obtained by adjusting the encoding parameters of the key frames; and the second bitstream is obtained by encoding the key point information extracted from the non-key frames of the video frames containing the target of interest by the first terminal.

17. A first terminal, characterized in that: include: a first processor and a first memory for storing a computer program executable on said first processor; Wherein, when the first processor is used to run the computer program, the steps of the method described in any one of claims 1 to 10 are executed.

18. A second terminal, characterized in that: include: a second processor and a second memory for storing a computer program executable on the second processor; Wherein, when the second processor is used to run the computer program, the steps of the method described in any one of claims 11 to 14 are executed.

19. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 10, or implements the steps of the method according to any one of claims 11 to 14.

20. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 10, or implements the steps of the method according to any one of claims 11 to 14.

Images