Video stream recovery method, electronic device and computer readable storage medium

Abstract

The embodiments of the present application disclose a video stream recovery method, an electronic device and a computer-readable storage medium, which are used to improve the effect of the recovered video stream, make the video stream clearer and smoother, and have a video picture that is closer to the real thing. The method includes: a first electronic device processes the original video stream to obtain a first video stream with a high frame rate and low resolution and a second video stream with a low frame rate and high resolution, and sends the first video stream and the second video stream to a second electronic device; the second electronic device determines the optical flow between the first target frame and the previous frame for each first target frame in the first video stream, and generates an approximate image frame according to the optical flow and the second target frame in the second video stream; generates a third target frame according to the approximate image frame and the second target frame; and generates a third video stream with a high frame rate and high resolution based on the second video stream and the third target frame.

Description

Technical Field

[0001] The present application relates to the field of multimedia technology, and in particular to a video stream recovery method, an electronic device, and a computer-readable storage medium. Background Art

[0002] With the continuous development of video technology, the application scenarios of video streaming are becoming more and more extensive, and the requirements for the resolution and frame rate of video streaming are becoming higher and higher.

[0003] The higher the resolution and / or frame rate of the video stream, the more network bandwidth is required. Usually, the network bandwidth between the sender and the receiver is limited, which limits the resolution and / or frame rate of the transmitted video stream, and the video clarity and / or smoothness of the receiver are low.

[0004] In order to resolve the contradiction between network bandwidth and the resolution and / or frame rate of the video stream, the receiving end can restore the video stream through super-resolution and frame insertion. The process of restoring the video stream through super-resolution is as follows: the receiving end processes the received low-resolution video stream through super-resolution to improve the resolution of the video frame and restore the high-resolution video stream. At this time, the sending end and the receiving end transmit the low-resolution video stream. The process of restoring the video stream through frame insertion is as follows: the receiving end estimates the intermediate frame based on the two existing video frames in the high-resolution and low-frame-rate video stream, and then inserts the intermediate frame to improve the frame rate, and restores the high-resolution and high-frame-rate video stream. At this time, the sending end and the receiving end transmit the high-resolution and low-frame-rate video stream. However, the video streams restored by these methods are of poor quality, and there are problems of blur or jitter in the moving picture. Summary of the invention

[0005] The embodiments of the present application provide a video stream recovery method, an electronic device, and a computer-readable storage medium, which can improve the effect of the recovered video stream, making the video stream clearer and smoother, and having a video picture that is closer to the real thing.

[0006] In a first aspect, an embodiment of the present application provides a video stream recovery method, which is applied to a second electronic device, and the method includes:

[0007] Receive a first video stream and a second video stream from a first electronic device; for each first target frame, determine the optical flow between the first target frame and the previous frame, the first target frame is a video frame corresponding to a first target moment in the first video stream, and the first target moment is a moment when there is a video frame in the first video stream and no video frame in the second video stream; for each first target frame, generate an approximate image frame according to the optical flow and the second target frame of the first target frame, the second target frame is a video frame corresponding to a second target moment in the second video stream, and the second target moment is a moment corresponding to a previous frame of the first target frame; for each first target frame, generate a third target frame according to the approximate image frame and the second target frame; generate a third video stream based on the second video stream and the third target frame.

[0008] Among them, the first video stream and the second video stream are both video streams obtained by processing the original video stream; the first video stream is a video stream with a first frame rate and a first resolution, the second video stream is a video stream with a second frame rate and a second resolution, the first frame rate is greater than the second frame rate, and the first resolution is less than the second resolution; the third video stream is a video stream with a third frame rate and a third resolution, the third frame rate is greater than the second frame rate and is less than or equal to the first frame rate, and the third resolution is greater than the first resolution and is less than or equal to the second resolution.

[0009] As can be seen from the above, the second electronic device first calculates the optical flow between the first target frame and the previous frame in the first video stream (i.e., the high frame rate and low resolution video stream); then generates an approximate image frame based on the optical flow and the second target frame in the second video stream (i.e., the low frame rate and high resolution video stream); then generates a third target frame based on the approximate image frame and the second target frame; finally, each third target frame is inserted into the corresponding position in the second video stream to restore the third video stream with a high frame rate and high resolution.

[0010] In this way, not only the high-frequency reference information provided by the high-resolution video stream is utilized, but also the structured information of the missing frames in the time domain provided by the high-frame rate video stream is utilized, thereby making the video images of the restored video stream closer to reality.

[0011] In some possible implementations of the first aspect, the process of determining the optical flow between the first target frame and the previous frame may include: obtaining an inter-frame motion vector between the first target frame and the previous frame from a video decoder, the video decoder being used to decode the received encoded video stream; and interpolating and sampling the inter-frame motion vector to obtain the optical flow between the first target frame and the previous frame.

[0012] In this implementation, the inter-frame motion vector between two previous and next video frames is obtained from the video decoder, and then the inter-frame motion vector is interpolated and sampled to obtain the optical flow, which greatly reduces the amount of calculation, takes less time, and has lower power consumption.

[0013] In some possible implementations of the first aspect, the process of determining the optical flow between the first target frame and the previous frame may include:

[0014] The first target frame and the previous frame are used as the images of the highest layer of the image pyramid, and downsampling is performed layer by layer from the highest layer downward to obtain the images of each layer of the image pyramid, wherein, except for the highest layer, each layer in the image pyramid includes an image obtained by downsampling the image of the previous layer; feature extraction is performed on the image of each layer to obtain feature information of each layer; the feature information of the lowest layer is input into the mesh block of the lowest layer to obtain the optical flow of the lowest layer output by the mesh block of the lowest layer; starting from the second to last layer of the image pyramid, the following steps are performed layer by layer upward: the feature information of the current layer and the optical flow upsampling result of the next layer are input into the mesh block of the current layer to obtain the optical flow of the current layer output by the mesh block; the optical flow output by the mesh block of the highest layer is upsampled to obtain the optical flow between the first target frame and the previous frame.

[0015] In this implementation, the optical flow is calculated based on the difference between the previous and next video frames, so that the calculated optical flow information is more sufficient.

[0016] In some possible implementations of the first aspect, the process of generating an approximate image frame according to the optical flow and the second target frame corresponding to the first target frame may include: based on the optical flow and the second target frame, by T1 1 (x,y)=T0(xx i ,yy i ) generates an approximate image frame.

[0017] Among them, x i =f 01 (x,y)[0],y i =f 01 (x,y)[1], T0 is the second target frame, T1 1 (x,y) is the approximate image frame, f 01 is the optical flow between the first target frame and its previous frame, and (x, y) refers to the pixel coordinates on the image.

[0018] In some possible implementations of the first aspect, if there is no corresponding video frame at the second target moment in the second video stream, the second target frame is a third target frame of the previous first target frame.

[0019] In some possible implementations of the first aspect, the process of generating the third target frame based on the approximate image frame and the second target frame includes: inputting the approximate image frame and the second target frame into the image optimization neural network to obtain the third target frame output by the image optimization neural network.

[0020] In some possible implementations of the first aspect, the image optimization neural network is a first image optimization neural network or a second image optimization neural network;

[0021] The number of layers of the second image optimization neural network is greater than the number of layers of the first image optimization neural network, and / or the number of nodes in each layer of the second image optimization neural network is greater than the number of nodes in each layer of the first image optimization neural network.

[0022] In this implementation, a more complex image optimization neural network is used to generate output picture frames, thereby solving the problem of insufficient optical flow information caused by using inter-frame motion vectors to calculate optical flow.

[0023] In some possible implementations of the first aspect, the method may further include: monitoring the network speed of the communication link, the communication link being a link between the first electronic device and the second electronic device; when the network speed is less than a first network speed threshold, sending network congestion indication information to the first electronic device, the network congestion indication information including the network speed, for instructing the first electronic device to reduce the sending frame rate of the second video stream according to the network speed.

[0024] Among them, the high-resolution second video stream accounts for a large proportion of the transmitted data volume, so the transmission frame rate of the second video stream can be reduced to significantly reduce the transmitted data volume, thereby reducing network congestion.

[0025] In this way, when the network condition is bad, by reducing the sending frame rate of the second video stream, network congestion is reduced and the user experience is improved while ensuring that the second electronic device can restore the third video stream.

[0026] In some possible implementations of the first aspect, after sending network congestion indication information to the first electronic device, the method may further include: if the network speed is greater than a second network speed threshold, sending network speed recovery indication information to the first electronic device, the network speed recovery indication information including the network speed, for instructing the first electronic device to increase the sending frame rate of the second video stream according to the network speed.

[0027] In this implementation, when the network speed gradually recovers, the frame rate of the second video stream can be gradually increased to make the video picture of the video stream restored on the second electronic device more realistic.

[0028] In some possible implementations of the first aspect, the method may further include: monitoring the network speed of the communication link, where the communication link is a link between the first electronic device and the second electronic device; when the network speed is less than a third network speed threshold, determining the n value based on the network speed; sending the n value to the first electronic device, where the n value is used to instruct the first electronic device to send the second video stream in a manner of sending 1 frame every n frames when sending the video stream, where n is a positive integer greater than or equal to 1.

[0029] In this implementation, the decision process of adjusting the frame rate according to the network speed is executed on the second electronic device side. Compared with executing it on the first electronic device side, this method can reduce the decision-making capability requirements on the video stream sending side.

[0030] In some possible implementations of the first aspect, the process of determining the n value based on the network speed may include: determining whether the network speed meets the adjustment conditions for changing the n value from the current value to the target value; if so, setting the n value to the target value; if not, not changing the n value.

[0031] The adjustment condition for changing the value of n from the current value to the target value is:

[0032] This method gradually changes the value of n according to the network speed v, that is, gradually reduces the frame rate of the second video stream, so that the frame rate of the second video stream will not be reduced too low all at once.

[0033] In some possible implementations of the first aspect, the process of determining the n value based on the network speed may include: determining a network speed interval into which the network speed falls; and using a numerical value corresponding to the network speed interval as the n value.

[0034] Among them, the corresponding relationship between the network speed range and the n value is as follows: when a / nb≤v

[0035] In some possible implementations of the first aspect, the process of generating the third video stream based on the second video stream and the third target frame may include: inserting the third target frame corresponding to each first target frame into the second video stream to obtain the third video stream.

[0036] In a second aspect, an embodiment of the present application provides a video stream recovery method, which is applied to a first electronic device, and the method includes:

[0037] ​​The original video stream is processed to obtain a first video stream and a second video stream, wherein the first video stream is a video stream with a first frame rate and a first resolution, and the second video stream is a video stream with a second frame rate and a second resolution, wherein the first frame rate is greater than the second frame rate, and the first resolution is less than the second resolution; the first video stream and the second video stream are sent to a second electronic device, wherein every frame of the second video stream is sent; network congestion indication information is received from the second electronic device, wherein the network congestion indication information includes a network speed of a communication link, wherein the communication link is a link between the first electronic device and the second electronic device; a value of n is determined according to the network speed; and the second video stream is sent to the second electronic device in a manner of sending one frame every n frames, wherein n is a positive integer greater than 1.

[0038] Among them, the high-resolution second video stream accounts for a large proportion of the transmitted data volume, so the transmission frame rate of the second video stream can be reduced to significantly reduce the transmitted data volume, thereby reducing network congestion.

[0039] In this way, when the network condition is bad, by reducing the sending frame rate of the second video stream, network congestion is reduced and the user experience is improved while ensuring that the second electronic device can restore the third video stream.

[0040] In some possible implementations of the second aspect, the above process of determining the n value based on the network speed may include: determining whether the network speed meets the adjustment conditions for changing the n value from the current value to the target value; if so, setting the n value to the target value; if not, not changing the n value.

[0041] The adjustment condition for changing the value of n from the current value to the target value is:

[0042] In some possible implementations of the second aspect, the process of determining the n value according to the network speed may include: determining a network speed interval into which the network speed falls; using a value corresponding to the network speed interval as the n value;

[0043] Among them, the corresponding relationship between the network speed range and the n value is as follows: when a / nb≤v

[0044] In a third aspect, an embodiment of the present application provides a video recovery system, which includes a first electronic device and a second electronic device, wherein the first electronic device is used to execute any method of the first aspect, and the second electronic device is used to execute any method of the second aspect.

[0045] ​​In a fourth aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method of any one of the first or second aspects described above is implemented.

[0046] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the method of any one of the first aspect or the second aspect mentioned above.

[0047] In a sixth aspect, an embodiment of the present application provides a chip system, the chip system includes a processor, the processor is coupled to a memory, and the processor executes a computer program stored in the memory to implement the method as described in any one of the first aspect or the second aspect. The chip system can be a single chip, or a chip module composed of multiple chips.

[0048] In a seventh aspect, an embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device executes the method described in any one of the first aspect or the second aspect above.

[0049] It can be understood that the beneficial effects of the second to seventh aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A schematic block diagram of a video stream recovery system provided in an embodiment of the present application;

[0051] Figure 2 A schematic block diagram of an interactive process of a video stream recovery method provided in an embodiment of the present application;

[0052] Figure 3 A schematic diagram of a first video stream and a second video stream provided in an embodiment of the present application;

[0053] Figure 4 A schematic diagram of calculating optical flow based on an image pyramid provided in an embodiment of the present application;

[0054] Figure 5 A schematic diagram of calculating optical flow based on inter-frame motion vectors provided in an embodiment of the present application;

[0055] Figure 6 Another schematic diagram of a first video stream and a second video stream provided in an embodiment of the present application;

[0056] Figure 7 A schematic diagram of a training data set generation process provided in an embodiment of the present application;

[0057] Figure 8 A schematic diagram of a video stream provided in an embodiment of the present application;

[0058] Fig. 9 A schematic block diagram of video stream recovery provided in an embodiment of the present application;

[0059] Fig.10 A schematic diagram of generating a third target frame provided in an embodiment of the present application;

[0060] Fig.11 Another schematic block diagram of video stream recovery provided in an embodiment of the present application;

[0061] Fig.12 Another schematic diagram of a third target frame provided in an embodiment of the present application;

[0062] Fig.13 A schematic diagram of sending a second video stream according to a network speed provided in an embodiment of the present application;

[0063] Fig.14 A schematic diagram of a cloud gaming scenario provided in an embodiment of the present application;

[0064] Fig.15 A schematic diagram of a screen projection scenario provided in an embodiment of the present application;

[0065] Fig.16 Another schematic structural diagram of the first electronic device 11 and the second electronic device 12 provided in the embodiment of the present application;

[0066] Fig.17 A schematic diagram of a video stream provided in an embodiment of the present application;

[0067] Fig.18 A schematic diagram of the hardware structure of an electronic device 1800 provided in an embodiment of the present application. DETAILED DESCRIPTION

[0068] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures and technologies are provided to facilitate a thorough understanding of the embodiments of the present application.

[0069] See also Figure 1 , which is a schematic block diagram of a video stream recovery system provided in an embodiment of the present application. The system may include a first electronic device 11 and a second electronic device 12, and the first electronic device 11 is communicatively connected with the second electronic device 12.

[0070] The first electronic device 11 is a transmitting device, and the second electronic device 12 is a receiving device.

[0071] The first electronic device 11 is used to process the original video stream to obtain the first video stream and the second video stream, and then send the first video stream and the second video stream to the second electronic device 12. Generally, the frame rate of the original video stream is greater than or equal to the maximum frame rate of the first video stream and the second video stream, and the resolution is greater than or equal to the maximum resolution of the first video stream and the second video stream.

[0072] The second electronic device 12 is used for restoring a third video stream according to the first video stream and the second video stream after receiving the first video stream and the second video stream from the first electronic device 11 .

[0073] The first video stream is a video stream with a first frame rate and a first resolution, and the second video stream is a video stream with a second frame rate and a second resolution. The first frame rate is greater than the second frame rate, and the first resolution is less than the second resolution. That is, the first video stream is a video stream with a high frame rate and a low resolution, and the second video stream is a video stream with a low frame rate and a high resolution.

[0074] The third video stream is a video stream with a third frame rate and a third resolution. The third frame rate is greater than the second frame rate and less than or equal to the first frame rate, and the third resolution is greater than the first resolution and less than or equal to the second resolution. Typically, the third frame rate is equal to the first frame rate, and the third resolution is equal to the second resolution.

[0075] The frame rate of the restored third video stream is less than or equal to the frame rate of the original video stream, and the resolution is less than or equal to the resolution of the original video stream.

[0076] Exemplarily, the first frame rate of the first video stream is 60fps, and the first resolution is 480x270. The second frame rate of the second video stream is 30fps, and the second resolution is 1920x1080. The third frame rate of the restored third video stream is 60fps, and the third resolution is 1920x1080. Among them, 60fps means that the number of frames per second (fps) transmitted by the picture is 60 frames. Similarly, 30fps means that the number of frames transmitted per second is 30 frames.

[0077] In other words, the second electronic device 12 can restore a high frame rate and high resolution video stream based on a high frame rate and low resolution video stream and a low frame rate and high resolution video stream.

[0078] In the embodiment of the present application, when restoring the video stream, the second electronic device 12 not only uses the structured information of the missing frames in the time domain provided by the high frame rate and low resolution video stream, but also uses the high frequency reference information provided by the low frame rate and high resolution video stream, so that the restored video stream is clearer and smoother, and the video picture is closer to reality. Figure 2 The video stream recovery process is described in detail.

[0079] See also Figure 2 , is a schematic block diagram of an interactive process of a video stream recovery method provided in an embodiment of the present application, and the method may include the following steps:

[0080] Step S201: The first electronic device 11 processes the original video stream to obtain a first video stream and a second video stream.

[0081] For example, the frame rate of the original video stream is 60fps and the resolution is 1920x1080. The original video stream is processed to obtain a first video stream with a frame rate of 60fps and a resolution of 480x270. The original video stream is processed to obtain a second video stream with a frame rate of 30fps and a resolution of 1920x1080.

[0082] Step S202: The first electronic device 11 sends a first video stream and a second video stream to the second electronic device 12.

[0083] Step S203: The second electronic device 12 calculates the optical flow between each first target frame in the first video stream and a previous frame.

[0084] It should be noted that the first target frame refers to the video frame corresponding to the first target moment in the first video stream. The first target moment refers to the moment when there is a video frame in the first video stream but no video frame in the second video stream. Usually, there is at least one first target frame in the first video stream, and at least one first target frame can constitute a first target frame set.

[0085] For example, see Figure 3 A schematic diagram of a first video stream and a second video stream is shown, as Figure 3 As shown, between 0 and tm, the first video stream includes video frame A1, video frame A2, video frame A3, ..., and video frame An in sequence, and the second video stream includes video frame B1, video frame B2, video frame B3, ..., and video frame Bk in sequence. Wherein, n and k are both positive integers greater than or equal to 1, and m is greater than or equal to 0.

[0086] like Figure 3 As shown, at time t1, the first video stream has video frame A1, and the second video stream also has video frame B1, so t1 is not the first target time; at time t2, the first video stream has video frame A2, and the second video stream does not have video frames, so t2 is the first target time; at time t3, the first video stream has video frame A3, and the second video stream does not have video frames, so t3 is the first target time. Similarly, it can be seen that time t4 and time t7 are not the first target time, and time t5 and t6 are the first target time.

[0087] Figure 3The first target frame set in the first video stream in includes the following video frames: video frame A2, video frame A3, video frame A5 and video frame A6. That is, video frame A2, video frame A3, video frame A5 and video frame A6 are all first target frames.

[0088] For each first target frame in the first target frame set, the previous frame refers to the previous video frame that is adjacent to the first target frame in time sequence. For example, the previous frame of video frame A2 is video frame A1, the previous frame of video frame A3 is video frame A2, the previous frame of video frame A5 is video frame A4, and the previous frame of video frame A6 is video frame A5.

[0089] After the second electronic device 12 determines the first target frame, it calculates the optical flow between the first target frame and a frame preceding the first target frame for each first target frame.

[0090] Optical flow refers to the flow of light. When a person observes a moving object, the scene of the object forms a series of continuously changing images on the retina of the person's eye. This series of continuously changing information continuously "flows" through the retina (i.e., the image plane), like a "flow" of light, so it is called optical flow. Optical flow expresses the changes in the image, contains information about the target's movement, and can be used to determine the target's movement.

[0091] The optical flow between two adjacent video frames can characterize the grayscale change trend of each pixel on the image, which can be regarded as the instantaneous speed of the grayscale pixel moving on the image plane. That is, the optical flow can characterize the instantaneous motion speed vector information of each pixel. The optical flow can be expressed as a two-dimensional vector, which is used to represent the movement amount of the pixel representing the same object (object) in the current video frame to the next video frame.

[0092] There are many ways to calculate the optical flow between two adjacent video frames, and two methods are listed below as examples.

[0093] Method 1: Calculate optical flow based on image pyramid.

[0094] For each first target frame, downsampling is first performed layer by layer based on the first target frame and a previous frame to generate an image pyramid; then, based on the image pyramid, an optical flow between the first target frame and a frame previous to the first target frame is calculated.

[0095] The image pyramid includes at least one layer, and the number of layers can be set according to actual needs.

[0096] Each layer in the image pyramid includes an image obtained by downsampling the image of the previous layer, and a net block. Usually, the highest layer (i.e., the first layer) of the image pyramid includes the first target frame and the previous frame of the first target frame; the first target frame in the highest layer and the previous frame of the first target frame are downsampled to obtain the image of the second layer. And so on, the images are downsampled layer by layer to obtain the images of each layer.

[0097] When downsampling an image layer by layer, feature information of each layer can be extracted simultaneously. For example, the first target frame of the highest layer and the frame before the first target frame are downsampled respectively to obtain an image of the second layer. At the same time, feature extraction is performed on the first target frame to obtain feature information corresponding to the first target frame, and feature extraction is performed on the frame before the first target frame to obtain feature information corresponding to the previous frame. At this time, the feature information of the highest layer includes feature information corresponding to the first target frame and feature information corresponding to the previous frame.

[0098] For the lowest level of the image pyramid, there is no need to downsample the image at the lowest level, but it is necessary to perform feature extraction on the image at the lowest level to obtain the feature information of the lowest level.

[0099] For example, see Figure 4 The schematic diagram of calculating optical flow based on the image pyramid is shown, and the image pyramid includes the first layer (the highest layer), the second layer, the third layer and the fourth layer (the lowest layer). Among them, the first layer includes video frame 411, video frame 421 and mesh block 431. Video frame 411 is a first target frame, and video frame 421 is the previous frame of the first target frame. The second layer includes image 412, image 422 and mesh block 432. The third layer includes image 413, image 423 and mesh block 433. The fourth layer includes image 414, image 424 and mesh block 434.

[0100] Downsampling is performed layer by layer from the highest layer downwards. First, video frame 411 is downsampled to obtain image 412, and video frame 421 is downsampled to obtain image 422; then, image 412 is downsampled to obtain image 413, and image 422 is downsampled to obtain image 423; finally, image 413 is downsampled to obtain image 414, and image 423 is downsampled to obtain image 424.

[0101] Feature extraction is performed on the images of each layer to obtain feature information of each layer. For example, for the fourth layer, feature extraction is performed on image 414 to obtain feature information of image 414. Feature extraction is performed on image 424 to obtain feature information of image 424; for the third layer, feature extraction is performed on images 413 and 423 respectively to obtain feature information of the third layer; for the second layer, feature extraction is performed on images 412 and 422 respectively to obtain feature information of the second layer; for the first layer, feature extraction is performed on video frames 411 and 421 respectively to obtain feature information of the first layer.

[0102] After extracting the feature information of each layer, from the lowest layer upwards, based on the feature information of each layer, the optical flow of each layer is calculated in sequence through the mesh blocks of each layer. The input of each mesh block is the feature information of the current layer and the optical flow upsampling result of the next layer, and the output is the optical flow of the current layer.

[0103] For example, Figure 4 As shown, first, the feature information of the image 414 and the feature information of the image 424 are input into the mesh block 434 to obtain the optical flow output by the mesh block 434, which is the optical flow of the fourth layer.

[0104] After calculating the optical flow of the fourth layer, the feature information of the image 413, the feature information of the image 423, and the up-sampling result of the optical flow of the fourth layer are input to the mesh block 433 to obtain the optical flow output by the mesh block 433, which is the optical flow of the third layer. The up-sampling result of the optical flow of the fourth layer refers to the result obtained by up-sampling the optical flow output by the mesh block 434.

[0105] After calculating the optical flow of the third layer, the feature information of the image 412, the feature information of the image 422, and the up-sampling result of the optical flow of the third layer are input to the network block 432 to obtain the optical flow output by the network block 432, which is the optical flow of the second layer. The up-sampling result of the optical flow of the third layer refers to the result obtained by up-sampling the optical flow output by the network 433.

[0106] Similarly, the feature information of the video frame 411, the feature information of the video frame 421 and the upsampling result of the optical flow of the second layer are input to the mesh block 431 to obtain the optical flow output by the mesh block 421, which is the optical flow of the first layer.

[0107] Finally, after obtaining the optical flow of the highest layer, the optical flow of the highest layer is upsampled to obtain the upsampled result of the highest layer, and the upsampled result of the highest layer is used as the optical flow between the first target frame and the previous frame. Figure 4 As shown, the optical flow output by the network block 431 is upsampled to obtain the optical flow f 01 , f 01 That is, the optical flow between the video frame 411 and the video frame 421.

[0108] The optical flow calculated in this way is pixel-level optical flow with finer granularity. Pixel-level optical flow refers to the optical flow in pixels.

[0109] Method 2: Calculate optical flow based on inter-frame motion vectors.

[0110] In other embodiments, for each first target frame, an inter-frame motion vector between the first target frame and a previous frame of the first target frame is first obtained from a video decoder, and then the inter-frame motion vector is interpolated and sampled to obtain an optical flow between the first target frame and a previous frame of the first target frame.

[0111] It should be noted that the inter-frame motion vector output by the video decoder is in blocks, with a coarse granularity. Each block includes multiple pixels. In order to obtain finer-grained optical flow information, the inter-frame motion vector can be interpolated and sampled to convert the coarse-grained inter-frame motion vector into a finer-grained pixel-level optical flow.

[0112] For example, see Figure 5 Schematic diagram of calculating optical flow based on inter-frame motion vectors, Figure 5 The arrows in represent optical flow, which has direction and magnitude and is a two-dimensional vector.

[0113] In comparison, the first method calculates the optical flow based on the difference between the previous and next video frames, which has a large amount of calculation, takes a long time, and consumes high power. The second method directly obtains the inter-frame motion vector between the previous and next video frames from the decoder, and then interpolates and samples the inter-frame motion vector to obtain the optical flow, which greatly reduces the amount of calculation, takes less time, and consumes less power.

[0114] Step S204: The second electronic device 12 generates, for each first target frame, an approximate image frame corresponding to the first target frame according to the optical flow between the first target frame and a previous frame, and the second target frame corresponding to the first target frame.

[0115] It should be noted that the second target frame corresponding to the first target frame refers to the video frame corresponding to the second target moment in the second video stream. The second target moment refers to the moment of the previous frame of the first target frame.

[0116] For each first target frame, if there is a video frame at the second target moment in the second video stream, the video frame corresponding to the second target moment is used as the second target frame; if there is no video frame at the second target moment in the second video stream, the third target frame corresponding to the previous first target frame of the first target frame is used as the second target frame.

[0117] For example, Figure 3As shown, in the second video stream, there is a video frame B1 at time t1, a video frame B2 at time t4, and a video frame B3 at time t7; while there are no corresponding video frames at time t2, time t3, time t5, and time t6.

[0118] At this time, the first target frame set includes the following first target frames: video frame A2, video frame A3, video frame A5 and video frame A6. For the first target frame A2, its previous frame is video frame A1, and the time of video frame A1 is t1, that is, t1 is the second target time. At time t1, the video frame in the second video stream is video frame B1, then the second target frame corresponding to video frame A2 is video frame B1. Similarly, the second target frame corresponding to video frame A5 is video frame B2. However, for video frames A3 and A6, there are no corresponding video frames in the second video stream.

[0119] The second target frame corresponding to the video frame A2 is the video frame B1. For the video frame A2, an approximate image frame corresponding to the video frame A2 is generated according to the video frame B1 in the second video stream and the optical flow between the video frame A2 and the video frame A1.

[0120] Similarly, the second target frame corresponding to video frame A3 is video frame B2. For video frame A5, an approximate image frame corresponding to video frame A5 is generated based on video frame B2 in the second video stream and the optical flow between video frame A5 and video frame A4.

[0121] However, there is no second target frame corresponding to video frame A3 in the second video stream. At this time, for video frame A3, its previous frame is video frame A2, and the moment of video frame A2 is t2, that is, moment t2 is the second target moment. At moment t2, there is no corresponding video frame in the second video stream. In this case, the third target frame corresponding to the previous first target frame is used as the second target frame corresponding to the current first target frame. At this time, the current first target frame is video frame A3, the previous first target frame is video frame A2, the third target frame corresponding to the previous first target frame is the third target frame corresponding to video frame A2, and the third target frame corresponding to video frame A2 is the second target frame corresponding to video frame A3.

[0122] The generation process of the third target frame corresponding to the video frame A2 can be as follows: first calculate the optical flow between the video frame A1 and the video frame A2, generate the approximate image frame corresponding to the video frame A2 according to the optical flow and the video frame B1, and then generate the third target frame according to the approximate image frame corresponding to the video frame A2 and the video frame B1. The third target frame is used as the video frame corresponding to the time t2 in the second video stream.

[0123] There is no second target frame corresponding to video frame A6 in the second video stream. At this time, for video frame A6, its previous frame is video frame A5, and the moment of video frame A5 is t5, that is, moment t5 is the second target moment. At moment t5, there is no corresponding video frame in the second video stream. At this time, the third target frame corresponding to video frame A5 is used as the second target frame corresponding to video frame A6. Specifically, the optical flow between video frame A4 and video frame A5 is first calculated, and the approximate image frame corresponding to video frame A5 is generated according to the optical flow and video frame B2, and then the third target frame is generated according to the approximate image frame corresponding to video frame A5 and video frame B2, and the third target frame is used as the video frame corresponding to moment t5 in the second video stream.

[0124] That is to say, if a first target frame does not have a corresponding second target frame in the second video stream, it is necessary to first generate a third target frame corresponding to the previous first target frame, and then use the third target frame corresponding to the previous first target frame as the second target frame corresponding to the current first target frame. The generation process and related introduction of the third target frame are as follows.

[0125] For example, see Figure 6 Another schematic diagram of the first video stream and the second video stream is shown, Figure 6 As shown, the third target frame corresponding to video frame A2 is video frame B11, the third target frame corresponding to video frame A3 is video frame B12, the third target frame corresponding to video frame A5 is video frame B21, and the third target frame corresponding to video frame A6 is video frame B22.

[0126] For video frame A2, its corresponding second target frame is video frame B1. According to the optical flow between video frame A1 and video frame A2, and video frame B1, the approximate image frame corresponding to video frame A2 is calculated; then, according to the approximate image frame corresponding to video frame A2 and video frame B1, the third target frame corresponding to video frame A2 (i.e., video frame B11) is calculated.

[0127] For video frame A3, its corresponding second target frame is video frame B11. First, the optical flow between video frame A2 and video frame A3 is calculated, and the approximate image frame corresponding to video frame A3 is calculated based on the optical flow and video frame B11; then, the third target frame corresponding to video frame A3 (i.e., video frame B12) is calculated based on the approximate image frame corresponding to video frame A3 and video frame B11.

[0128] For video frame A5, its corresponding second target frame is video frame B2. First, the optical flow between video frame A5 and video frame A4 is calculated, and the approximate image frame corresponding to video frame A5 is calculated based on the optical flow and video frame B2. Then, the third target frame corresponding to video frame A5 (i.e., video frame B21) is calculated based on the approximate image frame corresponding to video frame A5 and video frame B2.

[0129] For video frame A6, its corresponding second target frame is video frame B21. First, the optical flow between video frame A5 and video frame A6 is calculated, and the approximate image frame corresponding to video frame A6 is calculated based on the optical flow and video frame B21. Then, the third target frame corresponding to video frame A6 (i.e., video frame B22) is calculated based on the approximate image frame corresponding to video frame 6 and video frame B21.

[0130] For each first target frame, after determining the second target frame corresponding to the first target frame, the approximate image frame corresponding to the first target frame is calculated according to the second target frame and the optical flow.

[0131] In some embodiments, the approximate image frame may be calculated by the following equation 1.

[0132] T1 1 =W(T0,f 01 ) (1)

[0133] The expansion of formula 1 is shown in formula 2 below.

[0134] T1 1 (x,y)=T0(xx i ,yy i ) (2)

[0135] Among them, x i =f 01 (x,y)[0],y i =f 01 (x,y)[1]. T0 is the second target frame, T1 1 (x,y) is the approximate image frame, f 01 is the optical flow between the first target frame and its previous frame, and (x, y) refers to the pixel coordinates on the image.

[0136] Step S205 : The second electronic device 12 generates, for each first target frame, a third target frame according to the approximate image frame corresponding to the first target frame and the second target frame corresponding to the first target frame.

[0137] It should be noted that the approximate image frame corresponding to the first target frame may be relatively blurry, so it is not directly inserted into the second video stream, but is optimized according to the second target frame to obtain a clearer video frame.

[0138] Of course, the approximate image frame can also be directly used as the output picture frame, but this will make the effect of the restored video stream poor, affecting the user experience.

[0139] In some embodiments, an image optimization neural network may be used to process the second target frame and the approximate image frame to generate a third target frame. Specifically, for each first target frame, the approximate image frame corresponding to the first target frame and the second target frame corresponding to the first target frame are input into the image optimization neural network to obtain a video frame output by the image optimization neural network, and the video frame output by the image optimization neural network is used as the third target frame.

[0140] The image optimization neural network can be a multi-layer convolutional neural network, and the image optimization neural network is pre-trained.

[0141] Exemplarily, the training process of the image optimization neural network may be as follows: firstly generate a training data set, and then use the training data set to train the neural network to obtain the required image optimization neural network.

[0142] For example, see Figure 7 The schematic diagram of the training data set generation process shown in the figure, the original video stream includes video frame T0, video frame T1, video frame T2, video frame T3 and video frame T4. The second video stream includes video frame T0. The first video stream includes video frame S0, video frame S1, video frame S2, video frame S3 and video frame S4.

[0143] First, the first target frame in the first video stream is determined. At this time, the video frame S1, the video frame S2, the video frame S3 and the video frame S4 ​​are all the first target frames.

[0144] Then, for each first target frame, the optical flow between the first target frame and the previous frame is calculated, and an approximate image frame is generated according to the second target frame corresponding to the first target frame and the optical flow between the first target frame and the previous frame.

[0145] At this time, for video frame S1, its corresponding second target frame is video frame T0 in the second video stream. After calculating the optical flow between video frame S1 and video frame S0, an approximate image frame T11 corresponding to video frame S1 is generated according to the optical flow and video frame T0.

[0146] Similarly, an approximate image frame T21 corresponding to the video frame S2, an approximate image frame T31 corresponding to the video frame S3, and an approximate image frame T41 corresponding to the video frame S4 ​​are calculated.

[0147] It should be noted that, for a first target frame, if there is no corresponding second target frame in the second video stream, the corresponding video frame in the original video stream is used as the second target frame. For example, for video frame S2, assuming that there is no corresponding second target frame in the second video stream, video frame T1 in the original video stream is used as the second target frame.

[0148] Next, the video frame T0 and the approximate image frame T11 are used as input, and the video frame T1 in the original video stream is used as the corresponding true value. That is, the video frame T0 and the approximate image frame T11 are used as a set of data, and the video frame T1 is used as the true value corresponding to the set of data. In this way, based on the original video stream and a series of approximate image frames, multiple sets of data and the true values ​​corresponding to the multiple sets of data can be obtained, and then these multiple sets of data are combined into a training data set.

[0149] After obtaining the training data set, the neural network is trained using the training data set to obtain the required image optimization neural network. For example, the video frame T0 and the approximate image frame T11 are input into the neural network to obtain the output of the neural network; the difference between the output of the neural network and the video frame T1 (i.e., the true value) is then calculated, and the parameters of the neural network are adjusted according to the difference. In this way, the neural network is iteratively trained using the training data continuously until the loss value of the neural network tends to be stable, and the image optimization neural network is obtained.

[0150] Of course, in other embodiments, the approximate image frame may be generated by other methods, which are not limited here.

[0151] It is worth pointing out that since the inter-frame motion vector obtained from the video decoder has a coarse granularity and may only include unidirectional motion vectors, if the optical flow is calculated using the above method 2, there may be a problem of insufficient optical flow information. Insufficient optical flow information will affect the effect of the subsequently restored video frame, that is, the effect of the generated third target frame. For example, insufficient optical flow information may cause at least one of the following problems in the restored video: the restored video picture is unrealistic, the video picture is jittery, and the clarity is deteriorated.

[0152] In order to compensate for the problem of insufficient optical flow information, a more complex image optimization neural network can be used to generate the third target frame. A more complex image optimization neural network means that the network has more layers or more nodes in each layer.

[0153] Step S206: The second electronic device 12 generates a third video stream based on the second video stream and the third target frame.

[0154] After generating a third target frame corresponding to each first target frame, each third target frame may be inserted into a corresponding position in the second video stream to obtain a third video stream.

[0155] It can be understood that inserting each third target frame into the corresponding position in the second video stream can be regarded as restoring the frame rate of the second video stream, thereby obtaining a third video stream with high resolution and high frame rate.

[0156] For example, see Figure 8The schematic diagram of the video stream shown in the figure shows that the first video stream includes video frame A1, video frame A2, ..., video frame An. The second video stream includes video frame B1, video frame B2, ..., video frame Bk. The third video stream includes video frame B1, video frame B11, video frame B12, video frame B2, video frame B21, video frame B22, video frame B3, ..., video frame Bk.

[0157] based on Figure 6 It can be seen that video frame B11 is the third target frame corresponding to video frame A2, video frame B12 is the third target frame corresponding to video frame A3, video frame B21 is the third target frame corresponding to video frame A5, and video frame B22 is the third target frame corresponding to video frame A6.

[0158] Insert video frame B11 into the second video stream as the video frame corresponding to time t2 in the second video stream. Insert video frame B12 into the second video stream as the video frame corresponding to time t3 in the second video stream. Insert video frame B21 into the second video stream as the video frame corresponding to time t5 in the second video stream. Insert video frame B22 into the second video stream as the video frame corresponding to time t6 in the second video stream.

[0159] By analogy, the third target frames corresponding to all subsequent first target frames in the first video stream are calculated, and these third target frames are inserted into corresponding positions in the second video stream to obtain a third video stream.

[0160] As can be seen from the above, the second electronic device 12 first calculates the optical flow between the first target frame and the previous frame in the first video stream (i.e., a video stream with high frame rate and low resolution); then generates an approximate image frame based on the optical flow and the second target frame in the second video stream (i.e., a video stream with low frame rate and high resolution); then generates a third target frame based on the approximate image frame and the second target frame; finally, each third target frame is inserted into the corresponding position in the second video stream to restore a third video stream with high frame rate and high resolution.

[0161] In this way, not only the high-frequency reference information provided by the high-resolution video stream is utilized, but also the structured information of the missing frames in the time domain provided by the high-frame rate video stream is utilized, thereby making the video images of the restored video stream closer to reality.

[0162] In addition, by transmitting the first video stream and the second video stream instead of directly transmitting the original video stream, the frame rate and resolution of the restored video stream are higher under the same data transmission volume, and the restored video is clearer and smoother. In other words, under the conditions of equal clarity and / or equal frame rate, the bit rate can be greatly reduced.

[0163] Exemplarily, the embodiments of the present application can be viewed as utilizing a high frame rate and low resolution video stream to restore the frame rate of a low frame rate and high resolution video stream to obtain a high frame rate and high resolution video stream.

[0164] In order to increase the frame rate of the video stream, it can be achieved through interpolation. However, when generating the estimated intermediate frames, the existing interpolation method does not have the optical flow information provided by the high-frame-rate low-resolution video stream, and thus lacks the motion information between adjacent video frames, resulting in the problem of image misalignment in the estimated intermediate frames and unrealistic video images.

[0165] Compared with the existing interpolation method, the embodiment of the present application utilizes the optical flow between two adjacent video frames in the high-frame rate and low-resolution video stream when generating the interpolation frame of the low-frame rate and high-resolution video stream. The optical flow provides motion information between the video frames, thereby making the generated high-resolution video frame more realistic.

[0166] Of course, from another perspective, compared with the existing super-resolution method, the embodiment of the present application utilizes the high-frequency reference information of the high-resolution video frame provided by the low frame rate high-resolution video stream, and the restored video stream will not have problems such as blurred images.

[0167] As shown above, the embodiments of the present application exemplarily introduce two methods of calculating optical flow. The video stream recovery process under these two methods is exemplarily introduced below with reference to the accompanying drawings.

[0168] See also Fig. 9 A schematic block diagram of video stream recovery is shown, the original video stream includes video frame T0 and video frame T1, the second video stream includes video frame T0, the first video stream includes video frame S0 and video frame S1, and the third video stream includes video frame T0 and video frame T1 2 .

[0169] First, the first target frame in the first video stream is determined. At this time, at the moment corresponding to the video frame S1, there is no corresponding video frame in the second video stream, and the video frame S1 is the first target frame.

[0170] Then, for each first target frame, the optical flow between the first target frame and the previous frame is calculated. At this time, the previous frame of video frame S1 is video frame S0, so the optical flow f between video frame S1 and video frame S0 is calculated. 01 =F(S0, S1). The optical flow is calculated using the above method 1, which will not be described in detail here.

[0171] Next, an approximate image frame is generated according to the optical flow corresponding to the first target frame and the second target frame corresponding to the first target frame. At this time, the previous frame of the video frame S1 is the video frame S0, and the video frame corresponding to the video frame S0 is the video frame T0, then the video frame T0 is the second target frame corresponding to the video frame S1.

[0172] Specifically, through formula T1 1 =W(T0,f 01 ) calculates the approximate image frame T1 1 The specific process can be found above and will not be described here.

[0173] Finally, for each first target frame, a third target frame is generated according to the approximate image frame corresponding to the first target frame and the second target frame corresponding to the first target frame. 2 is the third target frame, namely T1 2 =C(T0, T1 1 ). According to the video frame T0 and the approximate image frame T1 1 , generate video frame T1 2 , and the video frame T1 2 Inserted into the second video stream.

[0174] For example, see Fig.10 A schematic diagram of generating a third target frame is shown, as Fig.10 As shown, video frame T0 and video frame T1 1 Input to the first image optimization neural network to obtain the video frame T1 output by the first image optimization neural network 2 .

[0175] See also Fig.11 Another schematic block diagram of video stream recovery is shown in FIG. Fig.11 As shown, firstly, the inter-frame motion vector between the video frame S1 and the video frame S0 is obtained from the video decoder, and then the inter-frame motion vector is interpolated and sampled to obtain the optical flow between the video frame S1 and the video frame S0. Then, according to the optical flow and video frame T0, calculate the approximate image frame T1 1 , and then according to the approximate image frame T1 1 And video frame T0, get video frame T1 2 .

[0176] in, Fig.11 and Fig. 9 The similarities or similarities will not be repeated here.

[0177] As can be seen from the above, the optical flow calculated based on the inter-frame motion vector may have the problem of insufficient optical flow information. In order to compensate for the optical flow information, a more complex image optimization neural network can be used to generate the third target frame. For example, see Fig.12 Another schematic diagram of the third target frame is shown, as Fig.11 As shown, video frame T0 and video frame T1 1Input to the second image optimization neural network to obtain the video frame T1 output by the second image optimization neural network 2 .

[0178] The granularity of the inter-frame motion vector is coarse, and there may be a problem of insufficient optical flow information. In order to compensate for the optical flow information, the second image optimization neural network has more layers and / or more nodes per layer than the first image optimization neural network. That is, the number of layers of the second image optimization neural network is more than the number of layers of the first image optimization neural network, or the number of nodes per layer in the second image optimization neural network is more than the number of nodes per layer in the first image optimization neural network, or the number of layers of the second image optimization neural network is more than the number of layers of the first image optimization neural network, and the number of nodes per layer in the second image optimization neural network is more than the number of nodes per layer in the first image optimization neural network.

[0179] Of course, at this time, the first image optimization neural network can also be used to generate a third target frame, but compared with the third target frame generated by the second image optimization neural network, the effect of the video restored by the former is poor.

[0180] In some embodiments, regardless of whether the network condition is good or bad, the frame rates of the first video stream and the second video stream remain unchanged. The frame rates of the two video streams remain unchanged, and the amount of data sent by the first electronic device 11 remains unchanged. If the network condition is not good, the first electronic device 11 does not reduce the amount of data sent, which is likely to cause network congestion or aggravate network congestion, affecting user experience.

[0181] When the network condition is not good, in order to reduce network congestion, the first electronic device 11 may reduce the amount of data sent.

[0182] Among the amount of data sent, the high-resolution second video stream accounts for a large proportion. Therefore, the amount of data sent can be significantly reduced by reducing the sending frame rate of the second video stream, thereby reducing network congestion. In addition, based on the video stream recovery method shown above, it can be seen that the frame rate of the recovered third video stream is related to the frame rate of the first video stream, but has nothing to do with the frame rate of the second video stream. In other words, even if the sending frame rate of the second video stream is reduced, it will not affect the frame rate of the recovered third video stream.

[0183] Based on this, in some other embodiments, the first electronic device 11 can reduce the sending frame rate of the second video stream according to the network v.

[0184] When adjusting the transmission frame rate of the second video stream, it is necessary to ensure that the second electronic device 12 can still restore the third video stream with the same frame rate after receiving the second video stream with adjusted frame rate. That is, the transmission frame rate of the second video stream cannot be reduced indefinitely.

[0185] For the second electronic device 12, although the first electronic device 11 reduces the sending frame rate of the second video stream in order to reduce network congestion, the second electronic device 12 can still restore the third video stream with the same frame rate after receiving the second video stream with the reduced frame rate. In this way, users can get a good user experience under different network conditions. In specific applications, the second electronic device 12 can monitor the current network speed v in real time, and determine whether network congestion occurs based on the monitored network speed v; if network congestion occurs, congestion information is fed back to the first electronic device 11, and the congestion information includes the current network speed v.

[0186] When determining whether network congestion occurs, the second electronic device 12 can determine in real time whether the current network speed v is less than or equal to a certain threshold value. If so, it is considered that network congestion occurs, and congestion information is fed back to the first electronic device 11. If not, it is considered that network congestion does not occur, and it continues to determine whether the current network speed is less than or equal to a certain threshold value. The threshold value can be set as needed and is not limited here.

[0187] Of course, the second electronic device 12 may also continuously feed back the monitored current network speed v to the second electronic device 12 , and the second electronic device 12 determines whether network congestion occurs based on the fed-back current network speed v.

[0188] That is to say, the network congestion determination process may be executed on the second electronic device 12 side, or on the first electronic device 11 side.

[0189] After determining that network congestion occurs, the first electronic device 11 can adopt the following two adjustment methods to reduce the sending frame rate of the second video stream: send the second video stream according to the n value corresponding to the network speed range in which the network speed v falls; and gradually reduce the sending frame rate of the second video stream according to the network speed v.

[0190] Adjustment method 1: Send the second video stream according to the n value corresponding to the network speed interval into which the network speed v falls.

[0191] In this manner, after network congestion occurs, the first electronic device 11 can first determine the network speed range into which the current network speed v falls, then determine the n value corresponding to the network speed range, and then send the second video stream according to the n value.

[0192] For example, when the network bandwidth is sufficient, that is, when there is no network congestion, the original bit rate of the original second video stream is a, and each frame of the second video stream is sent, that is, n=1.

[0193] Where n means: one frame is sent every n frames. For example, when n is 1, one frame is sent every 1 frame, that is, every frame is sent. When n=2, one frame is sent every 2 frames.

[0194] When network congestion occurs, the first electronic device 11 determines the network speed range in which the current network speed v falls and the corresponding n value according to the correspondence between the network speed range and the n value.

[0195] Exemplarily, the correspondence between the network speed range and the n value can be: when a / n - b ≤ v < a / (n - 1) - b, the n value is n. Specifically as follows:

[0196] When v ≥ a - b, n is 1 (that is, 1 frame is sent per 1 frame), and the bitrate is a at this time;

[0197] When a / 2 - b ≤ v < a - b, n is 2 (that is, 1 frame is sent per 2 frames), and the bitrate is approximately a / 2 at this time, and the frame rate is approximately 1 / 2 of the original frame rate;

[0198] When a / 3 - b ≤ v < a / 2 - b, n is 3 (that is, 1 frame is sent per 3 frames), and the bitrate is approximately a / 3 at this time, and the frame rate is approximately 1 / 3 of the original frame rate;

[0199] When a / 4 - b ≤ v < a / 3 - b, n is 4 (that is, 1 frame is sent per 4 frames), and the bitrate is approximately a / 4 at this time, and the frame rate is approximately 1 / 4 of the original frame rate;

[0200] When a / 5 - b ≤ v < a / 4 - b, n is 5 (that is, 1 frame is sent per 5 frames), and the bitrate is approximately a / 5 at this time, and the frame rate is approximately 1 / 5 of the original frame rate;

[0201] When n = the maximum value N, if v continues to decrease and v < v < a / (N - 1) - b, then the transmission of the second video stream is terminated.

[0202] Wherein, N means: without affecting the restoration of the video stream at the receiving end, the second video stream can be reduced to a maximum of 1 frame sent per N frames. b is a threshold value, which represents that a network speed reduction of b will affect the video stream (for example, video stuttering).

[0203] After the first electronic device 11 determines the n value corresponding to the current network speed v, it then sends the second video stream according to the n value.

[0204] For example, assume that the network speed range in which the current network speed v falls is a / 3 - b ≤ v < a / 2 - b, then it is determined that n = 3. That is, for the original second video stream, 1 frame is sent per 3 frames, or in other words, 1 frame is sent every 2 frames. At this time, the frame rate of the second video stream is approximately 1 / 3 of the original frame rate. Assume that the original frame rate of the original second video stream is 30fps, and the frame rate of the adjusted second video stream is approximately 10fps.

[0205] It should be noted that when adjusting the transmission frame rate of the second video stream using adjustment method 1, it may happen that the frame rate of the second video stream is suddenly reduced to a very low level. For example, when network congestion occurs for the first time and the network speed v falls within the range of a / 5 - b ≤ v < a / 4 - b, then n = 5 is determined. At this time, for the original second video stream, 1 frame is sent every 5 frames, or in other words, 1 frame is sent every 4 frames skipped, and the frame rate of the second video stream is approximately 1 / 5 of the original frame rate. It can also be seen from this that different from adjustment method 2, adjustment method 1 does not gradually reduce the transmission frame rate of the second video stream.

[0206] Adjustment method 2: Gradually reduce the transmission frame rate of the second video stream according to the network speed v.

[0207] In this method, the first electronic device 11 first determines whether the network speed v meets the adjustment condition for changing the n value from the current value to the target value. If the adjustment condition is met, the n value is changed from the current value to the target value to reduce the transmission frame rate of the second video stream by one level.

[0208] In specific applications, the transmission frame rate of the second video stream can be reduced by one level by changing the n value.

[0209] Exemplarily, the adjustment condition for changing the n value from the current value to the target value can be: v < a / (n - 1) - b, where the current value is n - 1 and the target value is n.

[0210] Specifically, the correspondence between the n value and the network speed v can be as follows:

[0211] When v < a - b, change n to 2 (i.e., send 1 frame every 2 frames). At this time, the bit rate is approximately a / 2 and the frame rate is approximately 1 / 2 of the original frame rate; a is the original bit rate, and the original bit rate is the bit rate of the original second video stream. At this time, the current value is 1 and the target value is 2. If v < a - b, then change the n value from 1 to 2, that is, n = 2.

[0212] When v < a / 2 - b, change n to 3 (i.e., send 1 frame every 3 frames). At this time, the bit rate is approximately a / 3 and the frame rate is approximately 1 / 3 of the original frame rate. At this time, the current value is 2 and the target value is 3. If v < a / 2 - b, then change the n value from 2 to 3, that is, n = 3.

[0213] When v < a / 3 - b, change n to 4 (i.e., send 1 frame every 4 frames). At this time, the bit rate is approximately a / 4 and the frame rate is approximately 1 / 4 of the original frame rate. At this time, the current value is 3 and the target value is 4. If v < a / 3 - b, then change the n value from 3 to 4, that is, n = 4.

[0214] When v < a / 4 - b, change n to 5 (i.e., send 1 frame every 5 frames). At this time, the bit rate is approximately a / 5, and the frame rate is approximately 1 / 5 of the original frame rate. At this time, the current value is 4, and the target value is 5. If v < a / 4 - b, then change the n value from 4 to 5, that is, n = 5.

[0215] When n = the maximum value N, if v continues to decrease and v < a / (N - 1) - b, then terminate the transmission of the second video stream.

[0216] When there is no network congestion, n = 1, that is, for the original second video stream, each frame is sent. It should be noted that the n value (i.e., send 1 frame every n frames) is for the original second video stream.

[0217] At a certain moment, the first network congestion occurs. The first electronic device 11 determines whether the current network speed v satisfies the adjustment condition. At this time, the adjustment condition is v < a - b, that is, if the current network speed v is less than a - b, it is considered to satisfy the adjustment condition, and then change the n value from 1 to 2 to reduce the transmission frame rate of the second video stream by one level, reducing the transmission frame rate of the second video stream to 1 / 2 of the original. Conversely, if the current network speed v is greater than a - b, it is considered not to satisfy the adjustment condition, and the n value is not changed, and the transmission frame rate of the second video stream is not reduced.

[0218] It should be noted that when the current network speed v is less than a - b, the network speed v may be much less than a - b. For example, the network speed v may be less than a / 4 - b. In the second adjustment method, still change n to 2, rather than changing the n value to 5 as shown in the first adjustment method above. In this way, even when the network speed v is very small during a certain network congestion, the frame rate of the second video stream will not be reduced to a very low level at one time.

[0219] After changing the n value from 1 to 2, the second network congestion occurs. The first electronic device 11 then determines whether the current network speed v satisfies the adjustment condition. At this time, the adjustment condition is v < a / 2 - b. If it is satisfied, then change the n value from 2 to 3.

[0220] And so on. If it is necessary to change the n value from 3 to 4, the adjustment condition is v < a / 3 - b. If it is necessary to change the n value from 4 to 5, the adjustment condition is v < a / 4 - b.

[0221] From the above, it can be obtained that when a network congestion occurs, first determine whether the network v satisfies the adjustment condition v < a / (n - 1) - b. If it is satisfied, then send 1 frame every n frames to reduce the transmission frame rate of the second video stream by one level.

[0222] For example, refer to Fig.13 The schematic diagram of sending the second video stream according to the network speed shown, such as Fig.13As shown, the original second video stream includes video frames B1 to B20.

[0223] At the initial moment, there is no network congestion, and the first electronic device sends video frame B1 and video frame B2 in sequence, that is, each frame in the second video stream is sent. Assuming that there is no network congestion at a later moment, the first electronic device will send video frame B3, video frame B4, video frame B5, ..., video frame B20 in sequence.

[0224] However, at a certain moment, the second electronic device 12 detects that network congestion occurs, and then feeds back congestion information to the first electronic device 11. At this time, the first network congestion occurs.

[0225] After receiving the congestion information, the first electronic device 11 determines that the network speed v in the congestion information is less than ab, and determines that the adjustment condition is met, and determines n=2, that is, 1 frame is sent every 2 frames, or 1 frame is sent every other frame. At this time, video frame B3 is not sent, but video frame B4 is sent every other frame. That is, for video frames B3 and video frames B4 in the original second video stream, after the frame rate is adjusted for the first time, only video frame B4 is sent, and video frame B3 is not sent.

[0226] After the first electronic device 11 responds to the congestion information of the first network congestion and before the second network congestion occurs, it always sends 1 frame every 2 frames. At this time, after sending video frame B4, video frame B6 is sent, that is, for video frame B5 and video frame B6 in the original second video stream, only video frame B6 is sent. Similarly, for video frame B7 and video frame B8 in the original second video stream, only video frame B8 is sent. For video frame B9 and video frame B10 in the original second video stream, only video frame B10 is sent. That is, during this period of time, the video frames sent in sequence are video frame B4, video frame B6, video frame B8 and video frame B10.

[0227] After sending the video frame B10 , the second electronic device 12 detects that the second network congestion occurs, and then feeds back the congestion information to the first electronic device 11 .

[0228] After receiving the congestion information, the first electronic device 11 determines that the current network speed v satisfies the adjustment condition v

[0229] ​It can be understood that by adjusting the value of n, the frame rate of the second video stream can be adjusted. For example, when processing the original video stream, the frame rate of the obtained original second video stream is 30fps, and the frame rate of the obtained first video stream is 60fps. At the initial moment, the network bandwidth is sufficient, and the first electronic device 11 sends the second video stream with a frame rate of 30fps and the first video stream with a frame rate of 60fps to the second electronic device 12. At a certain moment, network congestion occurs. The first electronic device 11 determines that the current network speed v in the congestion information satisfies the adjustment condition v < a - b, and then determines that n = 2. At this time, the frame rate of the second video stream is approximately 15fps, that is, at this time, the first electronic device 11 sends the second video stream with a frame rate of 15fps and the first video stream with a frame rate of 60fps to the second electronic device.

[0230] It should be noted that Fig.13 After the first network congestion, since the first electronic device 11 reduces the sending frame rate of the second video stream and reduces the amount of data transmitted on the link, the network speed v may gradually recover. After a period of time when the network speed recovers and network congestion is detected again, this network congestion can be defined as the second network congestion.

[0231] Of course, after the first network congestion, although the first electronic device 11 reduces the amount of data sent, the reduced amount of sent data is still large, resulting in continuous network congestion. At this time, the second electronic device 11 will continuously feedback congestion information to the first electronic device 11. The second electronic device 12 will respond to the congestion information every preset period. For example, it responds to the congestion information every 10 seconds. When responding to the congestion information for the second time, it can be determined that the second network congestion has occurred.

[0232] Among them, responding to the congestion information once means that the second electronic device 12 adjusts the sending frame rate of the second video stream according to the network speed v in the congestion information.

[0233] When network congestion occurs, the first electronic device 11 reduces the frame rate of the second video stream according to the current network speed v in the congestion information, reduces the amount of sent data, and reduces network congestion. This enables users to still obtain a good experience in a situation where the network condition is not good.

[0234] It should be noted that the decision-making process of determining the value of n according to the network speed v can be executed on the side of the first electronic device 11 or on the side of the second electronic device 12.

[0235] As described above, the second electronic device 12 monitors the network speed v in real time and determines whether network congestion occurs according to the network speed v. After network congestion occurs, it feeds back the congestion information to the first electronic device 11. The first electronic device 11 determines the corresponding value of n according to the network speed v in the congestion information.

[0236] In other embodiments, the second electronic device 12 does not feed back the congestion information to the first electronic device 11, but feeds back the n value to the first electronic device 11. At this time, when the second electronic device 12 determines that network congestion occurs according to the monitored network speed v, it determines the corresponding n value according to the network speed v, and feeds back the n value to the first electronic device 11. After receiving the n value, the first electronic device 11 sends the second video stream according to the n value.

[0237] Of course, the second electronic device 12 can also feed back the monitored network speed v to the first electronic device 11 in real time, and the first electronic device 11 determines whether network congestion occurs based on the network speed v, and after determining that network congestion occurs, determines the corresponding n value based on the network speed v.

[0238] It is understandable that when the first electronic device 11 adjusts the transmission frame rate of the second video stream according to the network speed v, the second electronic device 12 will receive the second video stream with a different frame rate. At this time, based on the video stream recovery method mentioned above, a third video stream with the same frame rate is recovered according to the second video stream and the first video stream with different frame rates.

[0239] For example, for the first electronic device 11, the frame rate of the second video stream sent at the initial moment is 30fps and the resolution is 1920x1080; the frame rate of the first video stream is 60fps and the resolution is 480x270. At a certain moment, network congestion occurs, and n=2 is determined, that is, the frame rate of the second video stream sent is approximately 15fps and the resolution is 1920x1080, and the first video stream remains unchanged.

[0240] For the second electronic device 11, during a period of time, the frame rate of the received second video stream is 30fps and the resolution is 1920x1080; the frame rate of the first video stream is 60fps and the resolution is 480x270. According to the video stream recovery method mentioned above, a third video stream is recovered, and the frame rate of the third video stream is 60fps and the resolution is 1920x1080.

[0241] During another period of time, the frame rate of the received second video stream is approximately 15fps and the resolution is 1920x1080; the frame rate of the first video stream is 60fps and the resolution is 480x270. According to the video stream recovery method mentioned above, a third video stream is recovered, and the frame rate of the third video stream is 60fps and the resolution is 1920x1080.

[0242] That is to say, even if the first electronic device 11 reduces the frame rate of the second video stream, the second electronic device 12 can still restore the third video stream with the same frame rate based on the second video stream with the reduced frame rate. In this way, even when network congestion occurs, the frame rate of the second video stream is reduced, the network congestion is reduced, and the video stream is smoother. In addition, although the frame rate of the second video stream is reduced, the effect of the restored video stream is still similar, and the user experience is high.

[0243] As described above, when network congestion occurs, the first electronic device 11 can gradually reduce the frame rate of the second video stream according to the network speed v. When the network speed is restored, the first electronic device 11 can also gradually increase the frame rate of the second video stream according to the network speed v.

[0244] At this time, the second electronic device 12 can feed back the monitored current network speed to the first electronic device 11 in real time, and the first electronic device 11 will determine whether the current network speed has risen to a certain threshold; or the second electronic device 12 can locally determine whether the monitored current network speed has risen to a certain threshold. If so, it is considered that the network speed has recovered, and network speed recovery indication information is fed back to the first electronic device 11, so that the first electronic device 11 can increase the frame rate of the second video stream according to the current network speed v in the network speed recovery indication information.

[0245] The process of increasing the frame rate of the second video stream according to the network speed v may be the opposite process of the process of reducing the frame rate of the second video stream according to the network speed v mentioned above. For example, at a certain moment, the network speed v1 fed back by the second electronic device 12 falls within the interval a / 4-b≤v

[0246] It should be noted that if the frame rate of the second video stream is too low, the video picture authenticity of the restored video stream may be poor. Therefore, when the network is congested, the frame rate of the second video stream can be appropriately or within a reasonable range to reduce the amount of transmitted data and reduce network congestion; and when the network speed gradually recovers, the frame rate of the second video stream can be gradually increased to make the video picture authenticity of the video stream restored by the second electronic device 12 better.

[0247] ​As can be seen from the above, the second electronic device 12 monitors whether network congestion occurs in real time, and when network congestion occurs, it feeds back network congestion information to the first electronic device 11. On the premise of ensuring that the second electronic device 12 can restore the third video stream based on the first video stream and the second video stream, the first electronic device 11 reduces the frame rate of the second video stream according to the current network speed in the network congestion information to reduce the amount of data sent, reduce network congestion, and provide a better user experience. In addition, when the network speed is restored, the frame rate of the second video stream is increased to improve the effect of the video picture of the restored video stream.

[0248] The following is an exemplary introduction to the video stream recovery solution provided in the embodiment of the present application in conjunction with specific scenarios.

[0249] See also Fig.14 , is a schematic diagram of a cloud gaming scenario provided in an embodiment of the present application. Fig.14 As shown, the first electronic device 11 is a cloud server 141, and the second electronic device 12 is a mobile phone 142. The mobile phone 142 and the cloud server 141 are connected to each other through a transmission medium 143, which can be exemplified by the Internet.

[0250] The cloud server 141 runs the game program and renders the game video screen, and then transmits the video stream of the game video screen to the mobile phone 142. After receiving the video stream, the mobile phone 142 decodes and renders the video stream and displays it on the display screen. The user can input the corresponding action through input devices such as keyboards and touch screens. After the mobile phone 141 collects these action information, it sends it to the cloud server 141. The cloud server 141 responds to these action information and renders a new game video screen, which is then transmitted to the mobile phone 141 through a video stream. In this way, the mobile phone 141 does not need to download and run the game program, and the user can also play games on the mobile phone 141.

[0251] At this time, the cloud server 141 first processes the video stream of the game video screen to obtain a first video stream with high frame rate and low resolution and a second video stream with low frame rate and high resolution, and sends the first video stream and the second video stream to the mobile phone 142.

[0252] After receiving the first video stream and the second video stream, the mobile phone 142 recovers a third video stream with a high frame rate and high resolution based on the video stream recovery method mentioned above, and displays the corresponding game screen based on the third video stream.

[0253] Furthermore, the mobile phone 142 can monitor the current network speed v in real time, and when network congestion occurs, feedback congestion information to the cloud server 141. After receiving the congestion information, the cloud server 141 reduces the frame rate of the second video stream according to the current network v to reduce the amount of data sent.

[0254] See also Fig.15 , is a schematic diagram of a screen projection scenario provided in an embodiment of the present application. Fig.15 As shown, the first electronic device 11 is a mobile phone 151, and the second electronic device 12 is a large-screen device 152. The mobile phone 151 and the large-screen device 152 can communicate via a Wi-Fi point-to-point connection.

[0255] After the mobile phone 151 establishes a Wi-Fi point-to-point connection with the large-screen device 152, the projection video stream is processed to obtain a first video stream with a high frame rate and low resolution and a second video stream with a low frame rate and high resolution, and the first video stream and the second video stream are sent to the large-screen device 152 via the Wi-Fi point-to-point connection.

[0256] After receiving the first video stream and the second video stream, the large-screen device 152 recovers a third video stream with a high frame rate and high resolution based on the video stream recovery method mentioned above, and displays the corresponding screen projection interface based on the third video stream.

[0257] Furthermore, the large-screen device 152 can monitor the current network speed v in real time, and when network congestion occurs, feedback congestion information to the mobile phone 151, and the mobile phone 151 reduces the frame rate of the second video stream according to the network speed v.

[0258] It should be noted that the cloud gaming scenario and screen projection scenario mentioned above are merely examples and are not intended to limit the actual scenarios of the embodiments of the present application.

[0259] See also Fig.16 , is another structural schematic diagram of the first electronic device 11 and the second electronic device 12 provided in the embodiment of the present application. Fig.16 As shown, the first electronic device 11 includes an application layer 111, a video encoder 112 and a network layer 113. The application layer 111 includes a dynamic frame rate adjustment module 1111 and a video stream sending module 1112. The second electronic device 12 includes an application layer 121, a video decoder 122 and a network layer 123. The application layer 121 includes a video stream recovery module 1211, and the network layer 123 includes a congestion monitoring module 1231.

[0260] The dynamic frame rate adjustment module 1111 is used to dynamically adjust the frame rate of the second video stream according to the current network speed v fed back by the second electronic device 12. The video stream sending module 1112 is used to send the first video stream and the second video stream to the second electronic device 12. The video encoder 112 is used to encode the video stream to be sent to obtain an encoded video stream. The network layer 113 is used to transmit the encoded video stream to the second electronic device 12.

[0261] The network layer 123 is used to receive the encoded video stream from the first electronic device 12. The video decoder 122 is used to decode the encoded video to obtain a video stream. The video stream recovery module 1211 is used to recover the third video stream according to the first video stream and the second video stream output by the video decoder 122. The congestion monitoring module 1231 is used to monitor the current network speed v in real time, and when it is determined that network congestion occurs, send congestion information to the first electronic device, and the congestion information carries the current network speed v.

[0262] In some other embodiments, the first electronic device 11 may not include the dynamic frame rate adjustment module 1111 , and the second electronic device 12 may not include the congestion monitoring module 1231 .

[0263] like Fig.16 As shown, after the original video stream is processed to obtain the first video stream 13 and the second video stream 14, the video stream sending module 1112 transmits the first video stream 13 and the second video stream 14 to the video encoder 112. The video encoder 112 encodes the first video stream 13 and the second video stream 14 to obtain the encoded first video stream 15 and the encoded second video stream 16.

[0264] The video encoder 112 passes the encoded video stream to the network layer 113. The network layer 113 processes the encoded first video stream 15 and the encoded second video stream 16 (for example, packetizing, etc.) to obtain a data stream 17, and then sends the transmission data stream 17 to the network layer 123 via the transmission medium 18.

[0265] After receiving the data stream 17 , the network layer 123 performs data parsing (eg, unpacking) on ​​the data stream 17 to obtain an encoded first video stream 15 and an encoded second video stream 16 , and transmits these video streams to the video decoder 122 .

[0266] The video decoder 122 decodes the encoded first video stream 15 and the encoded second video stream 16 respectively to obtain the first video stream 13 and the second video stream 14, and transmits the first video stream 13 and the second video stream 14 to the video recovery module 1211. The video recovery module 1211 recovers the third video stream 19 according to the first video stream 13 and the second video stream 14 based on the video recovery method mentioned above.

[0267] The congestion monitoring module 1231 of the second electronic device 12 monitors the current network speed v in real time. If it is determined that network congestion occurs based on the current network speed v, congestion information will be generated and transmitted to the first electronic device 11 via the transmission medium 18.

[0268] After receiving the congestion information, the dynamic frame rate adjustment module 1111 of the first electronic device 11 determines the corresponding n value according to the current network speed v carried in the congestion information, and according to the n value, reduces the sending frame rate of the second video stream 14 through the video stream sending module 1112. Of course, when the network speed v is restored, the dynamic frame rate adjustment module 1111 can also gradually increase the sending frame rate of the second video stream 14 to dynamically adjust the frame rate of the second video stream according to the network speed v.

[0269] It is understandable that the video encoding and decoding algorithm can be arbitrary. Take the H.264 algorithm as an example, see Fig.17 The video stream schematic diagram shown, the video streaming media (main) includes I frames and P frames, which are high frame rate and low resolution video streams, for example, it can be the encoded first video stream 15. The video streaming media (auxiliary) includes I frames and P frames, which are high resolution frame rate video streams, for example, it can be the encoded second video stream 16. According to these two video streams, the video stream recovered by the restored video stream is high frame rate and high resolution. It should be noted that the division of the above modules is only an example, and these modules can be hardware modules, software modules, or modules implemented by the joint action of software and hardware.

[0270] In the embodiment of the present application, the device types and specific structures of the first electronic device 11 and the second electronic device 12 are limited. For example, the first electronic device 11 or the second electronic device 12 can be a portable terminal device such as a mobile phone or a tablet computer, or a non-terminal device such as a server.

[0271] By way of example and not limitation, Fig.18 As shown, the electronic device 1800 may include a processor 1810, a memory 1820, a communication module 1830, and an antenna.

[0272] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 1800. In other embodiments of the present application, the electronic device 1800 may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware. For example, when the first electronic device 11 is a mobile phone, the electronic device 1800 may also include a display screen and a camera, etc.

[0273] The processor 1810 may include one or more processing units, for example, the processor 1810 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0274] The controller may be the nerve center and command center of the electronic device 1800. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0275] In some embodiments, the processor 1810 may include one or more interfaces, such as a universal asynchronous receiver / transmitter (UART) interface.

[0276] The UART interface is a universal serial data bus for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is generally used to connect the processor 1810 and the communication module 1830. For example, the processor 1810 communicates with the Bluetooth module in the communication module 1830 through the UART interface to implement the Bluetooth function, or communicates with the WiFi module in the communication module 1830 to implement the WiFi function.

[0277] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 1800. In other embodiments of the present application, the electronic device 1800 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0278] The communication function of the electronic device 1800 can be implemented through an antenna, a communication module 1830, a modem processor, and a baseband processor, etc. The antenna is used to transmit and receive electromagnetic wave signals.

[0279] The communication module 1830 may include a mobile communication module and / or a wireless communication module. The mobile communication module may provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the electronic device 1800. The wireless communication module may provide solutions for wireless communications including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), etc. applied to the electronic device 1800. The wireless communication module may be one or more devices integrating at least one communication processing module. The wireless communication module receives electromagnetic waves via an antenna, modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 1810. The wireless communication module may also receive the signal to be sent from the processor 1810, modulate the frequency of the signal, amplify it, and convert it into electromagnetic waves for radiation via the antenna.

[0280] The electronic device 1800 implements the display function through a GPU, a display screen, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 1810 may include one or more GPUs that execute program instructions to generate or change display information. For example, after the second electronic device 12 recovers the third video stream based on the first video stream and the second video stream, it displays the corresponding video screen through the GPU, the display screen, and the application processor.

[0281] The video codec includes a video encoder and a video decoder. The video encoder is used to compress digital video, and the video decoder is used to decompress digital video. The electronic device 1800 can support one or more video codecs. In this way, the electronic device 1800 can play or record videos in multiple coding formats, such as: moving picture experts group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0282] In the embodiment of the present application, the first electronic device 11 needs to use a video encoder to compress the video stream to be sent, and the second electronic device 12 needs to use a video decoder to decompress the received video stream.

[0283] The memory 1820 may include an internal memory and an external memory. The internal memory may be used to store computer executable program codes, and the executable program codes include instructions. The processor 1810 executes various functional applications and data processing of the electronic device 1800 by running the instructions stored in the internal memory. The internal memory may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 1800 (such as audio data, a phone book, etc.), etc. In addition, the internal memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0284] It should be noted that the hardware structures of the first electronic device 11 and the second electronic device 12 can refer to the electronic device 1800 .

[0285] Exemplarily, the first electronic device 11 includes a processor, a memory, a communication module and an antenna. The processor of the first electronic device 11 first processes the original video stream to obtain a first video stream and a second video stream, and then sends the first video stream and the second video stream to the second electronic device 12 through the communication module and the antenna.

[0286] The second electronic device includes a processor, a memory, a communication module and an antenna. The second electronic device can receive the first video stream and the second video stream through the antenna and the communication module, and then restore the third video stream through the processor.

[0287] The electronic device provided in the embodiment of the present application may include a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, a method such as any one of the above-mentioned video stream recovery method embodiments is implemented.

[0288] The embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0289] An embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0290] The present application also provides a chip system, which includes a processor coupled to a memory, and the processor executes a computer program stored in the memory to implement the methods described in the above method embodiments. The chip system can be a single chip or a chip module composed of multiple chips.

[0291] In the above-mentioned embodiments, the description of each embodiment has its own emphasis. For the part that is not described or recorded in detail in a certain embodiment, please refer to the relevant description of other embodiments. It should be understood that the size of the sequence number of each step in the above-mentioned embodiment does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. In addition, in the description of the present application specification and the attached claims, the terms "first", "second", "third", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance. The reference "one embodiment" or "some embodiments" described in the present application specification means that one or more embodiments of the present application include specific features, structures or characteristics described in combination with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in the different places in this specification are not necessarily all referring to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways.

[0292] Finally, it should be noted that the above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A video stream recovery method, characterized in that: Applied to a second electronic device, the method includes: receiving a first video stream and a second video stream from a first electronic device; For each first target frame, determine the optical flow between the first target frame and a previous frame, the first target frame being a video frame corresponding to a first target moment in the first video stream, the first target moment being a moment when a video frame exists in the first video stream and no video frame exists in the second video stream; For each of the first target frames, an approximate image frame is generated according to the optical flow and a second target frame of the first target frame, where the second target frame is a video frame corresponding to a second target moment in the second video stream, and the second target moment is a moment corresponding to a previous frame of the first target frame; For each of the first target frames, generating a third target frame according to the approximate image frame and the second target frame; Generate a third video stream based on the second video stream and the third target frame; Wherein, the first video stream and the second video stream are both video streams obtained by processing the original video stream; The first video stream is a video stream with a first frame rate and a first resolution, the second video stream is a video stream with a second frame rate and a second resolution, the first frame rate is greater than the second frame rate, and the first resolution is less than the second resolution; the third video stream is a video stream with a third frame rate and a third resolution, the third frame rate is greater than the second frame rate and less than or equal to the first frame rate, and the third resolution is greater than the first resolution and less than or equal to the second resolution; The step of generating an approximate image frame according to the optical flow and a second target frame corresponding to the first target frame comprises: Based on the optical flow and the second target frame, through T1 1 (x,y)=T0(xx i ,yy i ) generating the approximate image frame; Among them, x i =f 01 (x,y)[0],y i =f 01 (x,y)[1], T0 is the second target frame, T1 1 (x, y) is the approximate image frame, f 01 is the optical flow between the first target frame and the previous frame, and (x, y) refers to the pixel coordinates on the image; Generating a third target frame according to the approximate image frame and the second target frame, comprising: The approximate image frame and the second target frame are input into an image optimization neural network to obtain the third target frame output by the image optimization neural network.

2. The method according to claim 1, characterized in that Determining an optical flow between the first target frame and a previous frame includes: Obtaining an inter-frame motion vector between the first target frame and the previous frame from a video decoder, the video decoder being used to decode a received encoded video stream; Interpolation sampling is performed on the inter-frame motion vector to obtain an optical flow between the first target frame and the previous frame.

3. The method according to claim 1, characterized in that Determining an optical flow between the first target frame and a previous frame includes: Taking the first target frame and the previous frame as the highest layer images of the image pyramid, and performing downsampling layer by layer from the highest layer downwards to obtain images of each layer of the image pyramid, wherein, except for the highest layer, each layer in the image pyramid includes an image obtained by downsampling the image of the previous layer; Extract features from the images of each layer to obtain feature information of each layer; Inputting the feature information of the lowest layer into the mesh block of the lowest layer, and obtaining the optical flow of the lowest layer output by the mesh block of the lowest layer; Starting from the penultimate layer of the image pyramid, the following steps are performed layer by layer: inputting feature information of the current layer and the optical flow upsampling result of the next layer into the mesh block of the current layer to obtain the optical flow of the current layer output by the mesh block; The optical flow output by the mesh block of the highest layer is upsampled to obtain the optical flow between the first target frame and the previous frame.

4. The method according to claim 1, characterized in that If there is no corresponding video frame at the second target moment in the second video stream, the second target frame is the third target frame of the previous first target frame.

5. The method according to claim 1, characterized in that The image optimization neural network is a first image optimization neural network or a second image optimization neural network; The number of layers of the second image optimization neural network is greater than the number of layers of the first image optimization neural network, and / or the number of nodes in each layer of the second image optimization neural network is greater than the number of nodes in each layer of the first image optimization neural network.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: monitoring a network speed of a communication link, wherein the communication link is a link between the first electronic device and the second electronic device; When the network speed is less than a first network speed threshold, network congestion indication information is sent to the first electronic device, where the network congestion indication information includes the network speed and is used to instruct the first electronic device to reduce a sending frame rate of the second video stream according to the network speed.

7. The method according to claim 6, characterized in that The network congestion indication information is specifically used to instruct the first electronic device to determine the n value according to the network speed when sending the first video stream and the second video stream to the second electronic device, and when each frame of the second video stream is sent, and to send the second video stream to the second electronic device in a manner of sending one frame every n frames, where n is a positive integer greater than 1.

8. The method according to claim 7, characterized in that The network congestion indication information is used to instruct the first electronic device to determine whether the network speed satisfies an adjustment condition for changing the value of n from the current value to the target value; if so, the value of n is set to the target value; wherein the adjustment condition for changing the value of n from the current value to the target value is: v<(a / (n-1)-b), the current value is n-1, and the target value is n; v is the network speed, a is the original bit rate of the second video stream, and b is the network speed threshold; n is less than or equal to N, and n is greater than 1; Or it is used to instruct the first electronic device to determine the network speed range that the network speed falls into; the numerical value corresponding to the network speed range is used as the n value; wherein the correspondence between the network speed range and the n value is as follows: when (a / nb)≤v<(a / (n-1)-b), the n value is n; v is the network speed, a is the original bit rate of the second video stream, and b is the network speed threshold; n is less than or equal to N, and n is greater than 1.

9. The method according to claim 6, characterized in that After sending the network congestion indication information to the first electronic device, the method further includes: If the network speed is greater than a second network speed threshold, network speed recovery indication information is sent to the first electronic device, where the network speed recovery indication information includes the network speed and is used to instruct the first electronic device to increase the sending frame rate of the second video stream according to the network speed.

10. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: monitoring a network speed of a communication link, wherein the communication link is a link between the first electronic device and the second electronic device; When the network speed is less than a third network speed threshold, determining a value of n according to the network speed; The n value is sent to the first electronic device, where the n value is used to instruct the first electronic device to send the second video stream in a manner of sending one frame every n frames when sending the video stream, where n is a positive integer greater than or equal to 1.

11. The method according to claim 10, characterized in that Determining the value of n according to the network speed includes: Determine whether the network speed satisfies an adjustment condition for changing the value of n from a current value to a target value; If yes, set the value of n to the target value; The adjustment condition for changing the value of n from the current value to the target value is: v<(a / (n-1)-b), the current value is n-1, and the target value is n; v is the network speed, a is the original bit rate of the second video stream, b is the network speed threshold; n is less than or equal to N, and n is greater than 1.

12. The method according to claim 10, characterized in that Determining the value of n according to the network speed includes: Determine a network speed interval within which the network speed falls; The value corresponding to the network speed interval is used as the n value; The corresponding relationship between the network speed range and the n value is as follows: when (a / nb)≤v<(a / (n-1)-b), the n value is n; v is the network speed, a is the original bit rate of the second video stream, b is the network speed threshold; n is less than or equal to N, and n is greater than 1.

13. The method according to claim 1, characterized in that Generating a third video stream based on the second video stream and the third target frame, comprising: Insert the third target frame corresponding to each of the first target frames into the second video stream to obtain the third video stream.

14. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 13 is implemented.

15. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 13 is implemented.

Images