Video data processing method, device, equipment and storage medium

Abstract

The present disclosure provides a video data processing method, device, equipment and storage medium, and relates to the field of Internet technology. The specific implementation scheme includes: obtaining video data cached in a buffer area, the video data includes at least two video frames and an auxiliary information frame between each group of two adjacent video frames, the auxiliary information frame includes the picture change information of two adjacent video frames; according to the target auxiliary information frame and the Nth video frame, a video frame is determined from the 1st video frame to the Kth video frame as the target video frame, K is an integer greater than 1, N is an integer greater than K, the target auxiliary information frame includes all auxiliary information frames between the 1st video frame to the Kth video frame, and the picture difference between the target video frame and the Nth video frame meets the preset conditions; discard the video frames between the target video frame and the Nth video frame. The present disclosure can reduce the impact of frame loss on video coherence and improve the efficiency of frame loss.

Description

Technical Field

[0001] The present disclosure relates to the field of Internet technology, specifically to technical fields such as image processing, data transmission, and data encoding, and can be applied in the scenario of live video broadcasting, and in particular to video data processing methods, devices, equipment, and storage media. Background Art

[0002] Live video streaming has gradually become a mainstream form of expression on the Internet. Live video streaming may be affected by factors such as network quality or live streaming equipment quality, and may experience freezes and playback pauses. In order to ensure smooth transmission of live video, the live video receiver will increase the live video cache (i.e., pre-cache some video frame sequences) to deal with freezes and playback pauses. Live video is composed of multiple video frame sequences. When the cache of the receiving end continues to increase, the delay of the receiving end will continue to accumulate, which is called cumulative delay.

[0003] Current methods for reducing cumulative delay include directly discarding all video data buffered by the receiving end, or discarding one or two frames of the video data buffered by the receiving end.

[0004] However, directly discarding all video data cached by the receiving end will reduce the coherence of video playback. Discarding one or two frames of video data cached by the receiving end is inefficient. Summary of the invention

[0005] The present disclosure provides a video data processing method, apparatus, device and storage medium, which can reduce the impact of frame loss on video continuity, while improving the efficiency of frame loss and enhancing user experience.

[0006] According to a first aspect of the present disclosure, a video data processing method is provided, the method comprising:

[0007] The video data cached in the buffer area is obtained, where the video data includes at least two video frames and an auxiliary information frame between each group of two adjacent video frames, where the auxiliary information frame includes picture change information of the two adjacent video frames.

[0008] According to the target auxiliary information frame and the Nth video frame, a video frame is determined from the 1st video frame to the Kth video frame as the target video frame, K is an integer greater than 1, N is an integer greater than K, the target auxiliary information frame includes all auxiliary information frames between the 1st video frame to the Kth video frame, and the picture difference between the target video frame and the Nth video frame meets the preset conditions.

[0009] The video frames between the target video frame and the Nth video frame are discarded.

[0010] According to a second aspect of the present disclosure, a video data processing method is provided, the method comprising:

[0011] First video data to be sent is acquired, where the first video data includes at least two video frames.

[0012] An auxiliary information frame is inserted between each group of two adjacent video frames in the first video data to obtain second video data, wherein the auxiliary information frame includes picture change information of the two adjacent video frames.

[0013] The second video data is sent to the receiving end.

[0014] According to a third aspect of the present disclosure, a video frame processing device is provided, the device comprising: an acquisition unit, a determination unit, and a frame dropping unit.

[0015] The acquisition unit is used to acquire the video data cached in the buffer area, the video data includes at least two video frames and an auxiliary information frame between each group of two adjacent video frames, the auxiliary information frame includes picture change information of two adjacent video frames.

[0016] A determination unit is used to determine a video frame from the 1st video frame to the Kth video frame as a target video frame according to the target auxiliary information frame and the Nth video frame, K is an integer greater than 1, N is an integer greater than K, the target auxiliary information frame includes all auxiliary information frames between the 1st video frame to the Kth video frame, and the picture difference between the target video frame and the Nth video frame meets a preset condition.

[0017] The discarding unit is used to discard the video frames between the target video frame and the Nth video frame.

[0018] According to a fourth aspect of the present disclosure, a video frame processing device is provided, the device comprising: an acquisition unit, an insertion unit, and a sending unit.

[0019] The acquisition unit is used to acquire first video data to be sent, where the first video data includes at least two video frames.

[0020] The inserting unit is used to insert an auxiliary information frame between each group of two adjacent video frames in the first video data to obtain the second video data, wherein the auxiliary information frame includes picture change information of the two adjacent video frames.

[0021] The sending unit is used to send the second video data to the receiving end.

[0022] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a method as described in the first aspect or the second aspect.

[0023] According to a sixth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, where the computer instructions are used to cause a computer to execute the method according to the first aspect or the second aspect.

[0024] According to a seventh aspect of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the computer program implements the method according to the first aspect or the second aspect.

[0025] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0027] Figure 1 A schematic diagram of a flow chart of a video data processing method provided by an embodiment of the present disclosure;

[0028] Figure 2 The embodiments of the present disclosure provide Figure 1 A schematic diagram of an implementation process of S102;

[0029] Figure 3 The embodiments of the present disclosure provide Figure 2 A schematic diagram of an implementation process of S201;

[0030] Figure 4 Another schematic diagram of a flow chart of a video data processing method provided by an embodiment of the present disclosure;

[0031] Figure 5 A schematic diagram of the composition of a video data processing device provided in an embodiment of the present disclosure;

[0032] Figure 6 Another schematic diagram of the composition of the video data processing device provided by the embodiment of the present disclosure;

[0033] Figure 7 The present disclosure is provided as a schematic block diagram of an example electronic device 700 that can be used to implement the present disclosure. DETAILED DESCRIPTION

[0034] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0035] It should be understood that in the embodiments of the present disclosure, the character " / " generally indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", etc. are used only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features.

[0036] Live video streaming has gradually become a mainstream form of expression on the Internet. Live video streaming may be affected by factors such as network quality or live streaming equipment quality, and may experience freezes and playback pauses. In order to ensure smooth transmission of live video, the live video receiver will increase the live video cache (i.e., pre-cache some video frame sequences) to deal with freezes and playback pauses. Live video is composed of multiple video frame sequences. When the cache of the receiving end continues to increase, the delay of the receiving end will continue to accumulate, which is called cumulative delay.

[0037] For example, in a live video broadcast scenario, if the data transmission protocol is based on the transmission control protocol (TCP), such as the real-time messaging protocol (RTMP), HTTP-FLV, etc., HTTP-FLV means encapsulating the streaming media data into the FLV (flash video) format and then transmitting it through the hypertext transfer protocol (HTTP), then due to the reliable transmission characteristics of TCP, when freezes, playback pauses, etc. occur during the live broadcast, the delay on the viewing end will continue to accumulate, which is called cumulative delay.

[0038] For example, when a live video is playing in the foreground, the live video is playing and then retreats to the background. At this time, the player is paused and the audio and video data continues to be cached. When it returns to the foreground, it continues to play from the video stream data just exited, and the actual live broadcast is no longer at this time point. During this period of switching between the foreground and the background, a delay is accumulated. For another example, in the case of poor network or network jitter, in such a network environment, if there is no buffering strategy, the live broadcast will be stuck. In order to solve the problem of stuck, the playback end will increase the buffering strategy. Once buffering occurs, it means a delay from the push end (i.e. the sending end) to the playback end (i.e. the receiving end). When stuck situations occur multiple times, such delays will accumulate.

[0039] Current methods for reducing cumulative delay include directly discarding all video data buffered by the receiving end, or discarding one or two frames of the video data buffered by the receiving end.

[0040] However, directly discarding all video data cached by the receiving end will reduce the coherence of video playback. Discarding one or two frames of video data cached by the receiving end is inefficient.

[0041] Exemplarily, the current method of reducing cumulative delay includes directly discarding all video data cached by the receiving end, or discarding one or two frames of the video data cached by the receiving end. Directly discarding all video data cached by the receiving end has a greater impact on the continuity of the video, which is obvious to users and has a negative impact on the playback experience. The method of discarding one or two frames of video data cached by the receiving end has a limit on the amount of single frame loss, and the cumulative delay is eliminated more slowly, and the cumulative delay elimination efficiency is low.

[0042] Against this background, the present disclosure provides a video data processing method that can reduce the impact of frame loss on video continuity, while improving the efficiency of frame loss and enhancing user experience.

[0043] Exemplarily, the video data processing method provided by the embodiment of the present disclosure may be applied to a video data processing system, which may include a sending end, a server end, and a receiving end, wherein the sending end is connected to the server end, and the server end is connected to the receiving end.

[0044] Exemplarily, the execution subject of the video data processing method may be a receiving end of the above-mentioned video data processing system, wherein the receiving end may be a device with data processing capability, such as a mobile phone, a computer, etc. The execution subject of the method is not limited here.

[0045] In some embodiments, the server may be a single server, or a server cluster composed of multiple servers. In some implementations, the server cluster may also be a distributed cluster. The present disclosure does not limit the specific implementation of the server.

[0046] Figure 1 FIG. 1 is a flow chart of a video data processing method provided by an embodiment of the present disclosure. Figure 1 As shown, the method may include S101-S103.

[0047] S101, obtaining video data cached in a cache area.

[0048] Exemplarily, an auxiliary information frame may be inserted between each group of two adjacent video frames in the first video data to be sent at the sending end of the video data or the transit server of the video data (for example, the source station), and the auxiliary information frame includes the picture change information of the two adjacent video frames. The above-mentioned video data may include at least two video frames and an auxiliary information frame between each group of two adjacent video frames. The receiving end of the video data may receive the video data sent by the sending end to the receiving end. After receiving the video data sent by the sending end, the receiving end will cache the video data in the buffer area and play it.

[0049] S102 . Determine a video frame from the 1st video frame to the Kth video frame as a target video frame according to the target auxiliary information frame and the Nth video frame.

[0050] Exemplarily, one or more video frames whose picture difference with the Nth video frame meets the preset condition can be determined from the 1st video frame to the Kth video frame, and then a video frame is selected from the video frames meeting the preset condition as the target video frame, where K is an integer greater than 1, and N is an integer greater than K. After the video data cached in the buffer area is acquired, the target video frame whose picture difference with the Nth video frame meets the preset condition can be determined from the 1st video frame to the Kth video frame according to all auxiliary information frames between the 1st video frame to the Kth video frame in the video data, that is, the target auxiliary information frame.

[0051] For example, when K is 5 and N is 6, after obtaining the video data cached in the buffer area, the video frame that meets the preset condition can be determined from the first video frame to the fifth video frame according to all the auxiliary information frames between the first video frame to the fifth video frame in the video data, and then a video frame is selected from the video frames that meet the preset condition as the target video frame. In other words, when it is determined that the picture difference value between the third video frame and the sixth video frame meets the preset condition according to all the auxiliary information frames between the first video frame to the fifth video frame in the video data, the third video frame can be used as the target video frame.

[0052] S103: discard video frames between the target video frame and the Nth video frame.

[0053] Exemplarily, after the target video frame is determined, the video frames between the target video frame and the Nth video frame may be discarded.

[0054] For example, when K is 5 and N is 6, when it is determined that the third video frame in the video data is the target video frame, the fourth video frame and the fifth video frame may be discarded.

[0055] The present disclosure obtains the video data cached in the buffer area, then determines the target video frame from the 1st video frame to the Kth video frame according to the target auxiliary information frame between the 1st video frame to the Kth video frame in the video data, and finally discards the video frames between the target video frame to the Nth video frame. The influence of frame loss on video continuity can be reduced, while the efficiency of frame loss is improved, and the user experience is enhanced.

[0056] In some embodiments, before determining a video frame from the 1st video frame to the Kth video frame as the target video frame based on the target auxiliary information frame and the Nth video frame, the method may further include: determining that the data occupancy length of the video data is greater than a preset threshold.

[0057] Exemplarily, the data occupied length of the video data can be determined by the duration of the video data, or by the space occupied by the video data. When the data occupied length of the video data is determined by the duration of the video data, if the preset threshold is 10 seconds, then when the duration of the video data exceeds 10 seconds, it is considered that the data occupied length of the video data is greater than the preset threshold. When the data occupied length of the video data is determined by the space occupied by the video data, if the preset threshold is 5 gigabytes (GB), then when the space occupied by the video data exceeds 5GB, it is considered that the data occupied length of the video data is greater than the preset threshold.

[0058] This embodiment determines that the data occupied length of the video data is greater than a preset threshold value. When the data occupied length of the video data is greater than the preset threshold value, the video data can be subjected to frame dropping processing, thereby improving the accuracy of frame dropping of the video data and further improving the user experience.

[0059] In some embodiments, the data occupancy length of the 1st to Kth video frames and the target auxiliary information frame is less than or equal to a preset threshold, and the data occupancy length of the 1st to Nth video frames and the auxiliary information frames between the 1st to Nth video frames is greater than the preset threshold.

[0060] For example, when K is 5 and N is 6, the data occupancy length of the target auxiliary information frames from the 1st video frame to the 5th video frame and between the 1st video frame and the 5th video frame in the above video data is less than or equal to the preset threshold, and the data occupancy length of the auxiliary information frames from the 1st video frame to the 6th video frame and between the 1st video frame and the 6th video frame is greater than the preset threshold.

[0061] In this embodiment, by limiting the data occupied length of the 1st video frame to the Kth video frame and the target auxiliary information frame to be less than or equal to the preset threshold, and the data occupied length of the 1st video frame to the Nth video frame and the auxiliary information frame between the 1st video frame and the Nth video frame to be greater than the preset threshold, it can be ensured that the discarded video frame is in the cached video data, thereby improving the accuracy of frame loss.

[0062] In some embodiments, the data occupancy length of the auxiliary information frames from the 1st video frame to the K+1th video frame and from the 1st video frame to the K+1th video frame is greater than a preset threshold.

[0063] For example, when K is 5, it can be determined that the data occupied length of the auxiliary information frames from the 1st video frame to the 6th video frame and between the 1st video frame and the 6th video frame is greater than the preset threshold. The data occupied length of the auxiliary information frames from the 1st video frame to the 5th video frame and between the 1st video frame and the 5th video frame is less than or equal to the preset threshold. At this time, K can be understood as a critical value.

[0064] In this embodiment, by limiting the data occupation length of the auxiliary information frames from the 1st video frame to the K+1th video frame and from the 1st video frame to the K+1th video frame to be greater than a preset threshold, it can be determined that the video frames that can be discarded are selected from all the video frames in the video data whose data occupation length is greater than the preset threshold, thereby further improving the efficiency of frame dropping.

[0065] In some embodiments, the data occupation length of the auxiliary information frames from the 1st video frame to the N-1th video frame and between the 1st video frame and the N-1th video frame is less than or equal to a preset threshold.

[0066] Based on the above embodiment, for example, when N is 6, N-1 is 5, the data occupancy length of the auxiliary information frames from the 1st video frame to the 5th video frame and between the 1st video frame and the 5th video frame is less than or equal to the preset threshold, and the data occupancy length of the auxiliary information frames from the 1st video frame to the 6th video frame and between the 1st video frame and the 6th video frame is greater than the preset threshold, it can be determined that the 6th video frame is the first video frame after the above-mentioned cached video data.

[0067] This embodiment limits the data occupation length of the auxiliary information frames from the 1st video frame to the N-1th video frame and from the 1st video frame to the N-1th video frame to be less than or equal to a preset threshold. It can be determined that the Nth video frame is the first video frame after the above-mentioned cached video data, thereby improving the convenience of determining the discardable video frames.

[0068] In some embodiments, the preset condition may include: the picture difference value between the target video frame and the Nth video frame is the smallest.

[0069] Exemplarily, the picture difference value between each video frame in the video data cached in the buffer area and the Nth video frame can be calculated, and then the video frame with the smallest picture difference value with the Nth video frame can be determined as the target video frame, and the target video frame is a video frame that can be discarded.

[0070] In this embodiment, the preset condition includes that the image difference value between the target video frame and the Nth video frame is the smallest. The video frame with the smallest image difference value with the Nth video frame can be determined as the target video frame and discarded, further reducing the impact of frame loss on video continuity and improving the smoothness of video playback after frame loss.

[0071] Figure 2 The embodiments of the present disclosure provide Figure 1 A schematic diagram of an implementation flow of S102 is shown in FIG. Figure 2 As shown, Figure 1 S102 in the example may include S201 - S204 .

[0072] S201 . Determine a picture difference value between each group of two adjacent video frames between the first video frame to the Nth video frame according to an auxiliary information frame between each group of two adjacent video frames between the first video frame to the Nth video frame.

[0073] For example, the auxiliary information frame inserted between each group of two adjacent video frames includes the picture change information of the two video frames. A picture change calculation function can be represented by f, where p1 is a picture image, p2 is another picture image, and p3 is another picture image. For any three such picture images, f needs to satisfy: f(p1, p2) = f(p1, p3) + f(p3, p2).

[0074] S202 . Determine a picture difference value between the first video frame and the Nth video frame according to the picture difference value between each group of two adjacent video frames from the first video frame to the Nth video frame.

[0075] Exemplarily, the picture difference value between each group of two adjacent video frames between the first video frame and the Nth video frame may be determined based on the auxiliary information frame between each group of two adjacent video frames between the first video frame and the Nth video frame. Then, based on the picture difference value between each group of two adjacent video frames between the first video frame and the Nth video frame, the sum of the picture difference values ​​between each group of two adjacent video frames between the first video frame and the Nth video frame may be calculated to obtain the picture difference value between the first video frame and the Nth video frame.

[0076] For example, diff can be used to represent the image difference between two video frames, and diff_score can be used to represent the image difference value between two video frames. When N is 6, the image difference value between the 1st video frame and the Nth video frame is the sum of the image difference values ​​between each group of two adjacent video frames from the 1st video frame to the 6th video frame. The image difference value between the 1st video frame and the 6th video frame can be represented as: diff_score 16 =diff_score 12 +diff_score 23 +diff_score 34 +diff_score 45 +diff_score 56 Similarly, the image difference between the second video frame and the sixth video frame can be expressed as: diff_score 26 =diff_score 23 +diff_score 34 +diff_score 45 +diff_score 56 , the difference between the 3rd video frame and the 6th video frame is expressed as: diff_score 36 =diff_score 34 +diff_score 45 +diff_score 56 , the difference between the 4th video frame and the 6th video frame is expressed as: diff_score 46 =diff_score 45 +diff_score 56 , the image difference between the 5th video frame and the 6th video frame is expressed as: diff_score 56 .

[0077] S203 . Determine a picture difference value between the Kth video frame and the Nth video frame according to the picture difference value between each group of two adjacent video frames between the Kth video frame and the Nth video frame.

[0078] For example, the specific method for determining the picture difference value between the Kth video frame and the Nth video frame can refer to the specific method for determining the picture difference value between the 1st video frame and the Nth video frame, which will not be repeated here.

[0079] S204 , determining that the video frame with the smallest picture difference value with the Nth video frame among the 1st video frame to the Kth video frame is the target video frame.

[0080] For example, from the above embodiment, it can be known that K is an integer greater than 1, and N is an integer greater than K. After determining the picture difference value between each group of two adjacent video frames between the 1st video frame to the Nth video frame, the picture difference value between each video frame between the 1st video frame to the Nth video frame and the Nth video frame can be obtained. Then, from the 1st video frame to the Kth video frame, the video frame with the smallest picture difference value with the Nth video frame is determined as the target video frame.

[0081] This embodiment determines the picture difference value between each group of adjacent two video frames between the 1st video frame to the Nth video frame according to the auxiliary information frame between each group of adjacent two video frames between the 1st video frame to the Nth video frame; determines the picture difference value between the 1st video frame and the Nth video frame according to the picture difference value between each group of adjacent two video frames between the 1st video frame to the Nth video frame; determines the picture difference value between the Kth video frame and the Nth video frame according to the picture difference value between each group of adjacent two video frames between the Kth video frame to the Nth video frame; and can determine the video frame with the smallest picture difference value with the Nth video frame from the 1st video frame to the Kth video frame, and use the video frame as the target video frame, which can further reduce the impact of discarding the target video frame on the video coherence, and improve the video playback smoothness after frame loss.

[0082] Figure 3 The embodiments of the present disclosure provide Figure 2 A schematic diagram of an implementation process of S201 is shown in FIG. Figure 3 As shown, Figure 2 S201 in the example may include S301-S304.

[0083] S301 , determining a first matrix and a second matrix respectively corresponding to each group of two adjacent video frames between the 1st video frame to the Nth video frame.

[0084] Exemplarily, the first matrix and the second matrix are both p×q matrices.

[0085] For example, take two adjacent video frames as grayscale images p1 and p2. The grayscale image is composed of a number of pixels. The size of images p1 and p2 is p×q pixels. Then p1 can be represented as a matrix M1 of size p×q, M1 is the first matrix, and p2 can be represented as a matrix M2 of size p×q, M2 is the second matrix, where M1(i, j) is the grayscale value of the i-th row and j-th column of the first matrix. For images p1 and p2, the diff of p1 and p2 can be defined as: diff 12 =f(p1,p2)=M1–M2.

[0086] S302. Calculate the square of the difference between each element in the 0th row to the p-1th row and the 0th column to the q-1th column in the first matrix and each corresponding element in the second matrix to obtain p×q square results.

[0087] S303, calculating the sum of p×q square results to obtain a sum result.

[0088] S304 , calculating the ratio of the summation result to p×q, and obtaining the picture difference value between each group of two adjacent video frames between the 1st video frame to the Nth video frame.

[0089] For example, the picture difference value between each group of two adjacent video frames between the 1st video frame to the Nth video frame can be calculated according to a preset formula. When N is 2, the picture difference value between the 1st video frame and the 2nd video frame can be calculated according to formula (1).

[0090]

[0091] Among them, g is the picture difference value calculation function, [M1(i,j)-M2(i,j)] 2 represents p×q square results, Indicates the summation result, g(diff 12 ) represents the picture difference value between the first video frame and the second video frame, and p and q represent the pixels of the first video frame and the second video frame.

[0092] This embodiment determines the first matrix and the second matrix corresponding to each group of two adjacent video frames between the 1st video frame to the Nth video frame, and then calculates the square of the difference between each element in the 0th row to the p-1th row and the 0th column to the q-1th column in the first matrix and the corresponding element in the second matrix to obtain p×q square results; then calculates the sum of the p×q square results to obtain the sum result; finally, calculates the ratio of the sum result to p×q to obtain the picture difference value between each group of two adjacent video frames between the 1st video frame to the Nth video frame, so that the picture difference value between each group of two adjacent video frames between the 1st video frame to the Nth video frame can be calculated, so as to provide data support for the subsequent calculation of the picture difference value between each video frame between the 1st video frame to the Nth video frame and the Nth video frame.

[0093] The disclosed embodiment also provides a video data processing method, which is applied to a sending end or a server of the above-mentioned video frame processing system, wherein the sending end is used to send second video data to a receiving end, and the server is used to receive the second video data and forward the second video data to the receiving end.

[0094] Figure 4 Another flowchart of the video data processing method provided by the embodiment of the present disclosure is shown in FIG. Figure 4 As shown, the method may include S401-S403.

[0095] S401: Acquire first video data to be sent.

[0096] The first video data includes at least two video frames.

[0097] Exemplarily, the first video data is original video data to be sent by the sending end.

[0098] S402: insert an auxiliary information frame between each group of two adjacent video frames in the first video data to obtain second video data.

[0099] S403: Send the second video data to the receiving end.

[0100] Exemplarily, the second video data includes the first video data and an auxiliary information frame inserted between each group of two adjacent video frames in the first video data. The auxiliary information frame inserted between each group of two adjacent video frames includes picture change information of the two video frames. The picture change information of the two video frames may include: displacement or motion information, pixel value difference, image texture, optical flow, object segmentation information, etc. The displacement or motion information is the change of the position of the object between the two video frames, which can be expressed as a change such as translation, rotation, scaling or distortion of pixels. The pixel value difference is the change of the pixel value between the two video frames, that is, the difference in color or brightness. The image texture is the change of texture between the two video frames, such as edge, texture, texture direction, texture density, etc. The optical flow is the movement direction and speed information of the pixels between the two video frames. The object segmentation information is the segmentation information of the object between the two video frames, that is, the foreground and background in the video frame are separated. An auxiliary information frame including the picture change information of the two video frames can be inserted between each group of two adjacent video frames in the first video data to obtain the second video data, and then the second video data is sent to the receiving end. When receiving the second video data, the receiving end may cache part of the second video data in the cache area, that is, the video data cached in the cache area in the above embodiment.

[0101] For example, when the first video data includes video frame 1, video frame 2, video frame 3, video frame 4, video frame 5, and video frame 6, auxiliary information frame 1 can be inserted between video frame 1 and video frame 2, auxiliary information frame 2 can be inserted between video frame 2 and video frame 3, auxiliary information frame 3 can be inserted between video frame 3 and video frame 4, auxiliary information frame 4 can be inserted between video frame 4 and video frame 5, and auxiliary information frame 5 can be inserted between video frame 5 and video frame 6 to obtain second video data, that is, the second video data includes video frame 1, auxiliary information frame 1, video frame 2, auxiliary information frame 2, video frame 3, auxiliary information frame 3, video frame 4, auxiliary information frame 4, video frame 5, auxiliary information frame 5, and video frame 6, and then the second video data is sent to the receiving end.

[0102] This embodiment obtains the first video data to be sent, inserts an auxiliary information frame between each group of two adjacent video frames in the first video data, obtains the second video data, and then sends the second video data to the receiving end. This can reduce the impact of frame loss on video continuity, improve the efficiency of frame loss, and enhance user experience.

[0103] In some embodiments, the above method is applied to a sending device or a server, the sending device is used to send the second video data to the receiving device, and the server is used to receive the second video data and forward the second video data to the receiving end.

[0104] Exemplarily, the auxiliary information frame can be inserted between each group of two adjacent video frames in the first video data by an encoder of the first video data sending device, or the auxiliary information frame can be inserted between each group of two adjacent video frames in the first video data by a server. The sending device is used to send the second video data to the receiving device, and the server is used to receive the second video data and forward the second video data to the receiving end.

[0105] This embodiment limits the above method to be applied to a sending device or a server, the sending device is used to send the second video data to the receiving device, and the server is used to receive the second video data and forward the second video data to the receiving end. The richness of the method of inserting the auxiliary information frame between each group of two adjacent video frames in the first video data is increased, thereby ensuring that the auxiliary information frame is inserted between each group of two adjacent video frames in the video data received by the receiving end, providing data support for subsequent frame loss.

[0106] In an exemplary embodiment, the present disclosure also provides a video frame processing device, which can be used to implement the video data processing method implemented by the receiving end in the aforementioned embodiment. Figure 5 The following is a schematic diagram of the composition of the video frame processing device provided by the embodiment of the present disclosure. Figure 5 As shown, the device may include: an acquisition unit 501, a determination unit 502, and a discarding unit 503.

[0107] The acquisition unit 501 is used to acquire video data cached in the buffer area, where the video data includes at least two video frames and an auxiliary information frame between each group of two adjacent video frames, where the auxiliary information frame includes picture change information of two adjacent video frames.

[0108] The determination unit 502 is used to determine a video frame from the 1st video frame to the Kth video frame as the target video frame according to the target auxiliary information frame and the Nth video frame, K is an integer greater than 1, N is an integer greater than K, the target auxiliary information frame includes all auxiliary information frames between the 1st video frame to the Kth video frame, and the picture difference between the target video frame and the Nth video frame meets the preset condition.

[0109] The discarding unit 503 is configured to discard video frames between the target video frame and the Nth video frame.

[0110] Optionally, the determining unit 502 is further configured to determine whether the data occupancy length of the video data is greater than a preset threshold.

[0111] Optionally, the data occupancy length of the 1st to Kth video frames and the target auxiliary information frame is less than or equal to a preset threshold, and the data occupancy length of the 1st to Nth video frames and the auxiliary information frames between the 1st to Nth video frames is greater than the preset threshold.

[0112] Optionally, the data occupancy length of the auxiliary information frames from the 1st video frame to the K+1th video frame and from the 1st video frame to the K+1th video frame is greater than a preset threshold.

[0113] Optionally, the data occupancy length of the auxiliary information frames from the 1st video frame to the N-1th video frame and between the 1st video frame to the N-1th video frame is less than or equal to a preset threshold.

[0114] Optionally, the preset condition includes: a picture difference value between the target video frame and the Nth video frame is the smallest.

[0115] Optionally, the determination unit 502 is specifically used to determine the picture difference value between each group of adjacent two video frames between the 1st video frame to the Nth video frame according to the auxiliary information frame between each group of adjacent two video frames between the 1st video frame to the Nth video frame; determine the picture difference value between the 1st video frame and the Nth video frame according to the picture difference value between each group of adjacent two video frames between the 1st video frame to the Nth video frame; determine the picture difference value between the Kth video frame and the Nth video frame according to the picture difference value between each group of adjacent two video frames between the Kth video frame to the Nth video frame; and determine that the video frame with the smallest picture difference value with the Nth video frame among the 1st video frame to the Kth video frame is the target video frame.

[0116] Optionally, the determination unit 502 is specifically used to determine the first matrix and the second matrix corresponding to each group of two adjacent video frames between the 1st video frame to the Nth video frame, respectively, and the first matrix and the second matrix are both p×q matrices; calculate the square of the difference between each element in the 0th to p-1th row and the 0th to q-1th column in the first matrix and the corresponding element in the second matrix to obtain p×q square results; calculate the sum of the p×q square results to obtain a sum result; calculate the ratio of the sum result to p×q to obtain the picture difference value between each group of two adjacent video frames between the 1st video frame to the Nth video frame.

[0117] In an exemplary embodiment, the present disclosure also provides a video frame processing device, which can be used to implement the video data processing method implemented by the sending end or the server in the aforementioned embodiment. Figure 6 Another schematic diagram of the video frame processing device provided by the embodiment of the present disclosure. Figure 6 As shown, the device may include: an acquisition unit 601, an insertion unit 602, and a sending unit 603.

[0118] The acquisition unit 601 is configured to acquire first video data to be sent, where the first video data includes at least two video frames.

[0119] The inserting unit 602 is used to insert an auxiliary information frame between each group of two adjacent video frames in the first video data to obtain second video data, wherein the auxiliary information frame includes picture change information of two adjacent video frames.

[0120] The sending unit 603 is configured to send the second video data to the receiving end.

[0121] Optionally, the apparatus is applied to a sending device or a server, the sending device is used to send the second video data to the receiving device, and the server is used to receive the second video data and forward the second video data to the receiving end.

[0122] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0123] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0124] In an exemplary embodiment, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described in the above embodiments.

[0125] In an exemplary embodiment, the readable storage medium may be a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the method according to the above embodiments.

[0126] In an exemplary embodiment, a computer program product includes a computer program, and when the computer program is executed by a processor, the method according to the above embodiments is implemented.

[0127] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0128] like Figure 7 As shown, the electronic device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0129] Multiple components in the electronic device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the electronic device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0130] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as video data processing methods. For example, in some embodiments, the video data processing method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the video data processing method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the video data processing method in any other appropriate manner (e.g., by means of firmware).

[0131] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0132] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0133] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0134] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0135] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0136] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0137] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0138] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A video data processing method, the method comprising: Acquire video data cached in a buffer area, the video data including at least two video frames and an auxiliary information frame between each group of two adjacent video frames, the auxiliary information frame including picture change information of two adjacent video frames; According to the target auxiliary information frame and the Nth video frame, a video frame is determined from the 1st video frame to the Kth video frame as the target video frame, wherein K is an integer greater than 1, and N is an integer greater than K, the target auxiliary information frame includes all auxiliary information frames between the 1st video frame to the Kth video frame, and the picture difference between the target video frame and the Nth video frame meets a preset condition; the preset condition includes: the picture difference value between the target video frame and the Nth video frame is the smallest; The video frames between the target video frame and the Nth video frame are discarded.

2. The method according to claim 1, before determining a video frame from the 1st video frame to the Kth video frame as the target video frame according to the target auxiliary information frame and the Nth video frame, the method further comprises: It is determined that the data occupancy length of the video data is greater than a preset threshold.

3. According to the method of claim 2, the data occupancy length of the 1st video frame to the Kth video frame and the target auxiliary information frame is less than or equal to the preset threshold, and the data occupancy length of the 1st video frame to the Nth video frame and the auxiliary information frames between the 1st video frame and the Nth video frame is greater than the preset threshold. 4 . The method according to claim 3 , wherein the data occupancy length of the auxiliary information frames from the 1st video frame to the K+1th video frame and from the 1st video frame to the K+1th video frame is greater than the preset threshold.

5. According to the method of claim 3 or 4, the data occupancy length of the 1st video frame to the N-1th video frame and the auxiliary information frame between the 1st video frame to the N-1th video frame is less than or equal to the preset threshold.

6. The method according to claim 1, wherein determining a video frame from the 1st video frame to the Kth video frame as the target video frame according to the target auxiliary information frame and the Nth video frame comprises: Determine, according to the auxiliary information frame between each group of two adjacent video frames between the first video frame to the Nth video frame, a picture difference value between each group of two adjacent video frames between the first video frame to the Nth video frame; Determine a picture difference value between the first video frame and the Nth video frame according to a picture difference value between each group of two adjacent video frames between the first video frame and the Nth video frame; Determine a picture difference value between the Kth video frame and the Nth video frame according to a picture difference value between each group of two adjacent video frames between the Kth video frame and the Nth video frame; It is determined that the video frame with the smallest picture difference value with the Nth video frame among the 1st video frame to the Kth video frame is the target video frame.

7. The method according to claim 6, wherein determining the picture difference value between each group of adjacent two video frames between the first video frame to the Nth video frame according to the auxiliary information frame between each group of adjacent two video frames between the first video frame to the Nth video frame comprises: Determine a first matrix and a second matrix respectively corresponding to each group of two adjacent video frames between the first video frame and the Nth video frame, wherein the first matrix and the second matrix are both p×q matrices; Calculate the square of the difference between each element in the 0th row to the p-1th row and the 0th column to the q-1th column in the first matrix and each corresponding element in the second matrix to obtain p×q square results; Calculate the sum of the p×q square results to obtain a sum result; The ratio of the summation result to p×q is calculated to obtain the picture difference value between each group of two adjacent video frames between the first video frame to the Nth video frame.

8. A video data processing method, the method comprising: Acquire first video data to be sent, where the first video data includes at least two video frames; Inserting an auxiliary information frame between each group of two adjacent video frames in the first video data to obtain second video data, wherein the auxiliary information frame includes picture change information of two adjacent video frames; Send the second video data to a receiving end so that the receiving end executes the video data processing method according to any one of claims 1 to 7.

9. The method according to claim 8 is applied to a sending device or a server, the sending device is used to send the second video data to a receiving device, and the server is used to receive the second video data and forward the second video data to a receiving end.

10. A video data processing device, the device comprising: An acquisition unit, configured to acquire video data cached in a cache area, wherein the video data includes at least two video frames and an auxiliary information frame between each group of two adjacent video frames, wherein the auxiliary information frame includes picture change information between two adjacent video frames; a determination unit, configured to determine a video frame from the 1st video frame to the Kth video frame as a target video frame according to the target auxiliary information frame and the Nth video frame, wherein K is an integer greater than 1, and N is an integer greater than K, and the target auxiliary information frame includes all auxiliary information frames between the 1st video frame to the Kth video frame, and the picture difference between the target video frame and the Nth video frame meets a preset condition; the preset condition includes: the picture difference value between the target video frame and the Nth video frame is the smallest; The discarding unit is used to discard the video frames between the target video frame and the Nth video frame.

11. The apparatus according to claim 10, wherein the determining unit is further configured to: It is determined that the data occupancy length of the video data is greater than a preset threshold.

12. According to the device of claim 11, the data occupancy length of the 1st video frame to the Kth video frame and the target auxiliary information frame is less than or equal to the preset threshold, and the data occupancy length of the 1st video frame to the Nth video frame and the auxiliary information frame between the 1st video frame and the Nth video frame is greater than the preset threshold. 13 . The device according to claim 12 , wherein the data occupancy length of the auxiliary information frames from the 1st video frame to the K+1th video frame and from the 1st video frame to the K+1th video frame is greater than the preset threshold.

14. According to the device of claim 12 or 13, the data occupancy length of the 1st video frame to the N-1th video frame and the auxiliary information frames between the 1st video frame to the N-1th video frame is less than or equal to the preset threshold.

15. The device according to claim 10, wherein the determining unit is specifically configured to: Determine, according to the auxiliary information frame between each group of two adjacent video frames between the first video frame to the Nth video frame, a picture difference value between each group of two adjacent video frames between the first video frame to the Nth video frame; Determine a picture difference value between the first video frame and the Nth video frame according to a picture difference value between each group of two adjacent video frames between the first video frame and the Nth video frame; Determine a picture difference value between the Kth video frame and the Nth video frame according to a picture difference value between each group of two adjacent video frames between the Kth video frame and the Nth video frame; It is determined that the video frame with the smallest picture difference value with the Nth video frame among the 1st video frame to the Kth video frame is the target video frame.

16. The device according to claim 15, wherein the determining unit is specifically configured to: Determine a first matrix and a second matrix respectively corresponding to each group of two adjacent video frames between the first video frame and the Nth video frame, wherein the first matrix and the second matrix are both p×q matrices; Calculate the square of the difference between each element in the 0th row to the p-1th row and the 0th column to the q-1th column in the first matrix and each corresponding element in the second matrix to obtain p×q square results; Calculate the sum of the p×q square results to obtain a sum result; The ratio of the summation result to p×q is calculated to obtain the picture difference value between each group of two adjacent video frames between the first video frame to the Nth video frame.

17. A video data processing device, the device comprising: An acquisition unit, configured to acquire first video data to be sent, wherein the first video data includes at least two video frames; an inserting unit, configured to insert an auxiliary information frame between each group of two adjacent video frames in the first video data to obtain second video data, wherein the auxiliary information frame includes picture change information of two adjacent video frames; A sending unit, used for sending the second video data to a receiving end, so that the receiving end executes the video data processing method according to any one of claims 1-7.

18. The apparatus according to claim 17, wherein the apparatus is applied to a sending device or a server, wherein the sending device is used to send the second video data to a receiving device, and the server is used to receive the second video data and forward the second video data to a receiving end.

19. An electronic device comprising: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method of any one of claims 1 to 7, or the method of claim 8, or the method of claim 9.

20. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the method according to any one of claims 1 to 7, or the method according to claim 8, or the method according to claim 9.

21. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method according to any one of claims 1 to 7, or the method according to claim 8, or the method according to claim 9.

Images