Video rendering method and device, equipment and storage medium

Abstract

The embodiment of the invention discloses a video rendering method and device, equipment and a storage medium. According to the technical scheme provided by the embodiment of the invention, multiple paths of video streams are acquired, the latest video frame in each path of video stream is cached, and the currently cached latest video frame of each path of video stream is obtained; the currently cached video frames of each video stream are merged at preset processing time to obtain a target picture frame, and the target picture frame is rendered and displayed, and the preset processing time is determined by the target frame rate, so that the problem of relatively poor user experience in a multi-user video call process can be solved, the video call lagging condition is reduced, and the user experience is improved. And the user experience is improved.

Description

Technical Field

[0001] Embodiments of the present application relate to the technical field of video rendering, and in particular, to a video rendering method, apparatus, device, and storage medium. Background Art

[0002] With the continuous development of information technology, video calls have become an important part of people's daily work and life. During a video call, video data sent by a sending device is received and the video data is rendered and displayed.

[0003] During a multi-user video call, when multiple video data streams sent by multiple sending devices are received, the multiple video data streams need to be merged into a single video frame for rendering and display.

[0004] In related technologies, when a multi-user video call is made, since the network quality of each user may be different, there are certain differences in the transmission frame rates of the received video streams. When a receiving device renders a video frame, it usually directly merges all the video frames of multiple video streams. When the frame rate of a certain video stream is relatively high, it will lead to an increase in data storage, resulting in an increase in the overall data processing time, and causing stuttering during rendering and display, which affects the user experience. Summary of the Invention

[0005] Embodiments of the present application provide a video rendering method, apparatus, device, and storage medium, which can solve the problem of poor user experience during a multi-user video call, reduce video call stuttering, and improve the user experience.

[0006] In a first aspect, embodiments of the present application provide a video rendering method, including:

[0007] Obtain multiple video streams, cache the latest video frame in each video stream to obtain the currently cached video frame of each video stream;

[0008] At a preset processing time, merge the latest video frames currently cached in each video stream to obtain a target frame, and render and display the target frame. The preset processing time is determined by a target frame rate.

[0009] In a second aspect, embodiments of the present application provide a video rendering apparatus, including:

[0010] A video acquisition module, configured to obtain multiple video streams, cache the latest video frame in each video stream to obtain the currently cached video frame of each video stream;

[0011] A merging module, configured to merge the latest video frames currently cached in each video stream at a preset processing time to obtain a target picture frame, and render and display the target picture frame, where the preset processing time is determined by a target frame rate.

[0012] In a third aspect, an embodiment of the present application provides a video rendering device, including:

[0013] A memory and one or more processors;

[0014] The memory is used to store one or more programs;

[0015] When the one or more programs are executed by the one or more processors, the one or more processors implement the video rendering method as described in the first aspect.

[0016] In a fourth aspect, an embodiment of the present application provides a storage medium storing computer-executable instructions, and the computer-executable instructions are used to execute the video rendering method as described in the first aspect when executed by a computer processor.

[0017] In the embodiment of the present application, by obtaining multiple video streams, caching the latest video frames in each video stream, and at a preset processing time, merging the latest video frames currently cached in each video stream to obtain a target picture frame and rendering and displaying it. By adopting the above technical means, by caching the latest video frames in each video stream and only merging and outputting the latest video frames currently cached in each video stream, when the frame rate of a certain video stream is relatively high, it will not cause an increase in the data processing volume, thereby avoiding an increase in the data processing time, improving the working efficiency of the merged output of multiple video streams, and further reducing the stuttering situation in video calls and enhancing the user experience. Description of the Drawings

[0018] Figure 1 is a flowchart of a video rendering method provided by an embodiment of the present application;

[0019] Figure 2 is a schematic diagram of the frame rate comparison of each video stream provided by an embodiment of the present application;

[0020] Figure 3 is a schematic diagram of video frame caching provided by an embodiment of the present application;

[0021] Figure 4 is a schematic diagram of the rendering and display effect of a target picture frame provided by an embodiment of the present application;

[0022] Figure 5 is a schematic diagram of the rendering order of a target picture frame provided by an embodiment of the present application;

[0023] Figure 6 It is a schematic structural diagram of a video rendering device provided by an embodiment of the present application;

[0024] Figure 7 It is a schematic structural diagram of a video rendering device provided by an embodiment of the present application. Detailed implementation manners

[0025] In order to make the objectives, technical solutions and advantages of the present application clearer, the following further describes the specific embodiments of the present application in detail with reference to the accompanying drawings. It can be understood that the specific embodiments described herein are only used to explain the present application, rather than limiting the present application. Additionally, it should be noted that, for the sake of convenience of description, only parts related to the present application are shown in the accompanying drawings rather than all the content. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the operations (or steps) as sequential processes, many of the operations can be implemented in parallel, concurrently, or simultaneously. In addition, the order of the operations can be rearranged. When the operations are completed, the process can be terminated, but there may also be additional steps not included in the drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0026] The video rendering method, device, equipment and storage medium provided by the present application are intended to, during video rendering, cache the latest video frames in each video stream, and only merge and output the latest video frames currently cached in each video stream. When the frame rate of a certain video stream is relatively high, it will not cause an increase in the amount of data processing, can avoid an increase in data processing time, improve the working efficiency of merging and outputting each video stream, can reduce the lag situation in video calls, and enhance the user experience. Compared with the traditional video rendering method, it is usually affected by the video transmission frame rate. When conducting video calls among multiple users, since the network quality of each user is different, there are certain differences in the frame rates of the received video streams. When the receiving device renders the video picture, it usually directly merges and processes all the video frames of multiple video streams. When the frame rate of a certain video stream is relatively high, it will cause an increase in data storage, resulting in an increase in the overall data processing time and causing lag in the rendering display, affecting the user experience. Based on this, the video rendering method of the embodiments of the present application is provided to solve the problem of poor user experience in the existing video rendering process.

[0027] Figure 1The flowchart of a video rendering method provided by an embodiment of the present application is given. In this embodiment, the video rendering method can be executed by a video rendering device, which can be implemented in software and / or hardware. The video rendering device can be composed of two or more physical entities or one physical entity. Generally, the video rendering device can be an electronic device such as an interactive flat panel, a tablet computer, a smart TV, a learning machine, or a mobile phone.

[0028] The following takes an interactive flat panel as the main body for executing the video rendering method as an example for description. Refer to Figure 1 , the video rendering method specifically includes:

[0029] S101. Obtain multiple video streams, cache the latest video frame in each video stream, and obtain the currently cached video frame of each video stream.

[0030] During a video call, when multiple users participate in the video call, the receiving device will receive the video streams transmitted by all the electronic devices participating in the video call, that is, the receiving device can obtain multiple video streams. It should be noted that one of the multiple video streams is the video stream of the receiving device itself. Since each video stream comes from different users and the network quality of different users may also be different, there may be different transmission frame rates in the obtained video streams. To ensure that the various video streams with different frame rates can maintain a stable target frame rate when merged and output, the latest video frame in each video stream can be cached, so that subsequent merging and output processing can be performed according to the cached latest video frame, thereby avoiding an increase in the output time due to a large amount of data processing for the video stream with a higher input frame rate, reducing the actual number of frames processed for the video stream with a higher frame rate, improving the working efficiency of the merged output of various video streams, and further reducing the occurrence of jamming and enhancing the user experience.

[0031] Since each video stream comes from different users and the network quality of different users may also be different, there may be different input frame rates in the obtained video streams. Figure 2 is a schematic diagram of frame rate comparison of various video streams provided by an embodiment of the present application. Refer to Figure 2, in a certain video call scenario, assume that there are 4 users making a video call, then there are 4 video stream inputs. The input frame rates corresponding to each video stream are different. The input frame rate of video stream 1 is 60 FPS, the input frame rate of video stream 2 is 90 FPS, the input frame rate of video stream 3 is 30 FPS, and the input frame rate of video stream 4 is 20 FPS. Among them, FPS represents the number of video frames transmitted per second. Therefore, video stream 1 corresponds to 60 video frames input per second, video stream 2 corresponds to 90 video frames input per second, video stream 3 corresponds to 30 video frames input per second, and video stream 4 corresponds to 20 video frames input per second. After the receiving device obtains each video stream, it needs to output through a preset target frame rate. For example, if the target frame rate for outputting the synthesized video frame is set to 30 FPS, then 30 synthesized video frames need to be output per second, that is, about one synthesized video frame is output every 33 milliseconds. For video stream 1, the number of video frames input every 33 milliseconds is about 2 frames. For video stream 2, the number of video frames input every 33 milliseconds is about 3 frames. For video stream 3, the number of video frames input every 33 milliseconds is about 1 frame. For video stream 4, the number of video frames input every 33 milliseconds is 0.66 frames, that is, the number of video frames received in the first 33 milliseconds may be 1 frame, and the number of video frames received in the second 33 milliseconds is 1 frame; or the number of video frames received in the first 33 milliseconds may be 0 frames, and the number of video frames received in the second 33 milliseconds is 2 frames.

[0032] It should be noted that in the actual transmission process of each video stream, the number of frames received by the same video stream in different preset processing time periods (such as 33 milliseconds) is not the same. The number of frames received in each of the above preset processing time periods is only for illustrative purposes.

[0033] It should be noted that the preset processing time period can be understood as the time interval for outputting the synthesized video frame according to the target frame rate, and the preset processing time can be understood as the time point corresponding to the output of the synthesized video frame according to the target frame rate. Each of the above 33 milliseconds can be understood as a preset processing time period, and the corresponding time points of each 33 milliseconds can be understood as the preset processing time. For example, 33 milliseconds, 66 milliseconds, and 99 milliseconds are all preset processing times, and 0 - 33 milliseconds, 33 - 66 milliseconds, and 66 - 99 milliseconds are all preset processing time periods.

[0034] As described above, within a preset processing time period, such as 33 to 66 milliseconds, it is possible that the input frame number of a certain video stream is zero. For example, for the aforementioned video stream 4, in the related art, it is usually necessary to wait until a new video frame is received before performing the merge output processing. That is, it is equivalent that the corresponding preset processing time (i.e., the 66th millisecond) cannot perform video output, and it is necessary to wait until the next preset processing time period, that is, 66 to 99 milliseconds, until a new video frame is received for the corresponding video stream before the merge processing can be performed, which may lead to video stuttering. To solve this problem of video stuttering, this embodiment proposes that when receiving multiple video streams, the latest video frames in each video stream are cached, so that subsequent merge output processing can be performed based on the cached latest video frames. When no new video frames are received in the corresponding preset processing time period, the merge output can also be performed based on the latest video frames cached previously, avoiding stuttering and improving the user experience.

[0035] In one embodiment, when obtaining multiple video streams, for each video stream, the obtained latest video frame replaces the originally cached video frame for caching to obtain the currently cached video frame. Figure 3 This is the first schematic diagram of video frame caching provided by the embodiment of the present application. Refer to Figure 3 , within a certain preset processing time period, for video stream 1, within this preset processing time period, video frame A1 and video frame A2 are sequentially received, and video frame A0 was cached in the previous preset processing time period. Within this preset processing time period, when video frame A1 is received, video frame A1 replaces the cached video frame A0 for caching processing, and the currently cached video frame corresponding to video stream 1 is video frame A1. When video frame A2 is received, video frame A2 replaces the cached video frame A1 for caching processing, and the currently cached video frame corresponding to video stream 1 is video frame A2. As described above, by replacing the originally cached video frame with the latest video frame received for each video stream for caching processing, only one video frame is cached for each video stream within each preset processing time period, thereby reducing the data processing volume, shortening the data processing duration, and further reducing the occurrence of video rendering stuttering and improving the user experience.

[0036] Exemplarily, assume that the preset processing time period is 33 to 66 milliseconds. For video stream 1, within 33 to 66 milliseconds, video frames A1 and A2 are successively received. Corresponding to the previous preset processing time period, that is, 0 to 33 milliseconds, video frame A0 is cached. Within 33 to 66 milliseconds, when video frame A1 is received, video frame A1 replaces the cached video frame A0 for caching processing, and the currently cached video frame corresponding to video stream 1 is video frame A1. When video frame A2 is received, video frame A2 replaces the cached video frame A1 for caching processing, and the currently cached video frame corresponding to video stream 1 is video frame A2. Therefore, at the preset processing time, that is, at the 66th millisecond, the latest video frame currently cached for the corresponding video stream 1 is video frame A2.

[0037] It should be noted that for each video stream, only one video frame is cached.

[0038] In one embodiment, taking each preset processing time period as a unit of processing time, multiple video streams are continuously acquired. For each video stream, within the preset processing time period, if at least two video frames are received, the video frames of this video stream received within the preset processing time period are subjected to frame dropping processing to obtain the latest video frame of this video stream within the preset processing time period. It should be noted that the corresponding data processing time point is the end time of the corresponding preset processing time period, that is, the corresponding preset processing time (point). The latest video frame and the prior video frame within the preset processing time period can be determined according to the timestamps of the acquired video frames, the prior video frame is deleted, and the latest video frame within the preset processing time period is retained. The latest video frame within the preset processing time period is cached to obtain the currently cached video frame of each video stream.

[0039] It should be noted that the latest video frame within the preset processing time period can be directly cached, or the latest video frame within the preset processing time period can replace the original cached video frame for caching. By replacing the original cached video frame to cache the latest video frame within the preset processing time period, the overall data volume of data caching can be reduced, thereby improving the overall working efficiency of data processing.

[0040] Refer to Figure 3, within a certain preset processing time period, taking video stream 1 and video stream 2 as examples for illustration. For video stream 1, within this preset processing time period, video frames A1 and A2 are received in sequence, and video frame A0 was cached in the previous preset processing time period. Based on the sequential receipt of video frames A1 and A2 within this preset processing time period, at the preset processing time, according to the reception timestamps of video frames A1 and A2, it can be determined that video frame A2 is the latest video frame of video stream 1 received within this preset processing time period, while video frame A1 is the earlier received video frame of video stream 1 within this preset processing time period. Video frame A1 is deleted, and video frame A2 is retained as the latest video frame of video stream 1 within this preset processing time period. Video frame A2 replaces video frame A0 for caching, and the currently cached video frame corresponding to video stream 1 is video frame A2. For video stream 2, within this preset processing time period, video frames B1, B2, and B3 are received in sequence, and video frame B0 was cached in the previous preset processing time period. Based on the sequential receipt of video frames B1, B2, and B3 within this preset processing time period, at the preset processing time, according to the reception timestamps of video frames B1, B2, and B3, it can be determined that video frame B3 is the latest video frame of video stream 2 received within this preset processing time period, while video frames B1 and B2 are the earlier received video frames of video stream 2 within this preset processing time period. Video frames B1 and B2 are deleted, and video frame B3 is retained as the latest video frame of video stream 2 within this preset processing time period. Video frame B3 replaces video frame B0 for caching, and the currently cached video frame corresponding to video stream 2 is video frame B3. As described above, by performing replacement caching processing on the video frames received by each video stream, only one video frame is cached for each video stream within each preset processing time period, thereby reducing the data processing volume, shortening the data processing duration, and further reducing the occurrence of video rendering stuttering, improving the user experience.

[0041] Exemplarily, assume that the preset processing time period is 33 to 66 milliseconds. For video stream 2, within 33 to 66 milliseconds, video frames B1, B2, and B3 are received in sequence. Corresponding to the previous preset processing time period, i.e., within 0 to 33 milliseconds, video frame B0 is cached. Within 33 to 66 milliseconds, based on the sequentially received video frames B1, B2, and B3, for example, the timestamps of receiving video frames B1, B2, and B3 are the 43rd millisecond, the 54th millisecond, and the 65th millisecond respectively. According to the timestamps of receiving video frames B1, B2, and B3, it can be determined that video frame B3 is the latest video frame of video stream 2 received within 33 to 66 milliseconds, while video frames B1 and B2 are the prior video frames of video stream 2 received within 33 to 66 milliseconds. Video frames B1 and B2 are deleted, and video frame B3 is retained as the latest video frame of video stream 2 within 33 to 66 milliseconds. Video frame B3 is used to replace video frame B0 for caching, and the currently cached video frame corresponding to video stream 2 is video frame B3.

[0042] In one embodiment, when the input frame rate of the video stream is the same as the preset target frame rate, that is, the number of video frames received in each preset processing time period for this video stream is one frame. For example, for video stream 3, the received video frame is taken as the latest video frame of this video stream, and the latest video frame is cached or the latest video frame is used to replace the originally cached video frame for caching to obtain the currently cached video frame. Refer to Figure 3 , within a certain preset processing time period, taking video stream 3 as an example for illustration. For video stream 3, within this preset processing time period, video frame C1 is received. Corresponding to the previous preset processing time period, video frame C0 is cached. Based on only receiving video frame C1 within this preset processing time period, video frame C1 is taken as the latest video frame of video stream 3 within this preset processing time period. At the preset processing time, video frame C1 is cached or video frame C1 is used to replace video frame C0 for caching, and the currently cached video frame corresponding to video stream 3 is video frame C1. As described above, by caching the latest video frame received for each video stream, only one video frame is cached for each video stream corresponding to each preset processing time period, thereby reducing the data processing amount, shortening the data processing duration, and further reducing the occurrence of video rendering stuttering and improving the user experience.

[0043] Exemplarily, assume that the preset processing time period is 33 to 66 milliseconds. For video stream 3, within 33 to 66 milliseconds, video frame C1 is received. Corresponding to the previous preset processing time period, i.e., 0 to 33 milliseconds, video frame C0 is cached. At 33 to 66 milliseconds, based on only receiving video frame C1, video frame C1 is used as the latest video frame of video stream 3 at 33 to 66 milliseconds. Video frame C1 is used to replace video frame C0 for caching processing, and the currently cached video frame corresponding to video stream 3 is video frame C1.

[0044] In one embodiment, when the transmission frame rate of a certain video stream is low, it is possible that no video frame is received within a certain preset processing time period. Then, the originally cached video frame is continued to be cached to obtain the currently cached video frame of this video stream. Refer to Figure 3 , within a certain preset processing time period, taking video stream 4 as an example for illustration. For video stream 4, within this preset processing time period, no video frame is received. Corresponding to the previous preset processing time period, video frame D0 is cached. Based on not receiving a video frame within this preset processing time period, video frame D0 is continued to be cached, and the currently cached video frame corresponding to video stream 4 is video frame D0. In the next preset processing time period, if no video frame is still received, video frame D0 is continued to be cached, and the currently cached video frame corresponding to video stream 4 is video frame D0. In the next preset processing time period, if video frame D1 is received, video frame D1 is used to replace video frame D0 for caching, and the currently cached video frame corresponding to video stream 4 is video frame D1. As described above, by performing replacement caching processing on each video frame received by each video stream, only one video frame is cached corresponding to each video stream within each preset processing time period, thereby reducing the data processing amount and shortening the data processing duration, and further reducing the occurrence of video rendering jitter and improving the user experience.

[0045] Exemplarily, assume that the preset processing time period is 33 to 66 milliseconds. For video stream 4, within 33 to 66 milliseconds, no video frame is received. Corresponding to the previous preset processing time period, i.e., 0 to 33 milliseconds, video frame D0 is cached. At 33 to 66 milliseconds, based on not receiving a video frame, video frame D0 is continued to be cached, and the currently cached video frame corresponding to video stream 4 is video frame D0.

[0046] S102. At the preset processing time, the currently cached latest video frames of each video stream are merged to obtain a target picture frame, and the target picture frame is rendered and displayed. The preset processing time is determined by the target frame rate.

[0047] The preset processing time period can be understood as the time interval for outputting synthesized video frames according to the target frame rate, and the preset processing time can be understood as the time point corresponding to the output of synthesized video frames according to the target frame rate. Therefore, the length of the preset processing time period and the corresponding preset processing time (point) are determined by the target frame rate of video output. For example, when the target frame rate of output is set to 30 FPS, 30 synthesized video frames (i.e., target picture frames) need to be output per second, that is, one synthesized video frame is output approximately every 33 milliseconds. Then, the corresponding preset processing time period can be determined to be 33 milliseconds. That is, each 33 - millisecond period can be understood as a preset processing time period, and the time points of each 33 - millisecond period can be understood as preset processing times. For example, 33 milliseconds, 66 milliseconds, and 99 milliseconds are all preset processing times, and the periods from 0 to 33 milliseconds, 33 to 66 milliseconds, and 66 to 99 milliseconds are all preset processing time periods. At the preset processing time, the latest video frames of each video stream within the preset processing time period are cached to obtain the currently cached video frames of each video stream. At the corresponding preset processing time (point), the currently cached latest video frames of each video stream are merged according to the preset distribution area of each video stream on the canvas to obtain the target picture frame, and the target picture frame is rendered and displayed.

[0048] Exemplarily, Figure 4 is a schematic diagram of the rendering and display effect of the target picture frame provided by an embodiment of the present application. Refer to Figure 4 , assume that there are 4 users having a video call, namely local user 1, remote user 1, remote user 2, and remote user 3. Among them, local user 1 corresponds to video stream 1, remote user 1 corresponds to video stream 2, remote user 2 corresponds to video stream 3, and remote user 3 corresponds to video stream 4. The preset distribution of the canvas is that the video picture corresponding to local user 1 is located in area 10 of the canvas, the video picture corresponding to remote user 1 is located in area 20 of the canvas, the video picture corresponding to remote user 2 is located in area 30 of the canvas, and the video picture corresponding to remote user 3 is located in area 40 of the canvas. Assume that the preset processing time period is from 33 to 66 milliseconds. From the above, it can be obtained that the currently cached video frames corresponding to video streams 1 - 4 within 33 - 66 milliseconds are video frame A2, video frame B3, video frame C1, and video frame D0 respectively. At the preset processing time, that is, 66 milliseconds, the currently cached video frames A2, B3, C1, and D0 of video streams 1 - 4 are merged according to the preset distribution area of each video stream on the canvas to obtain the target picture frame corresponding to 66 milliseconds, and the target picture frame is rendered and displayed to obtain the video call display picture of the receiving device at the corresponding moment.

[0049] During a multi - user video call, the video picture corresponding to each user is displayed at the preset distribution position on the canvas. Figure 5It is a schematic diagram of the rendering order of target screen frames provided by an embodiment of the present application. Refer to Figure 5 , assuming that the synthesis processing of video frames for three preset processing times has been performed. The first target screen frame is obtained through synthesis processing at the first preset processing time (for example, the 33rd millisecond). The first target screen frame includes the first frame of the local user, the first frame of remote user 1, the first frame of remote user 2, and the first frame of remote user 3. The second target screen frame is obtained through synthesis processing at the second preset processing time (for example, the 66th millisecond). The second target screen frame includes the second frame of the local user, the second frame of remote user 1, the second frame of remote user 2, and the second frame of remote user 3. The third target screen frame is obtained through synthesis processing at the third preset processing time (for example, the 99th millisecond). The third target screen frame includes the third frame of the local user, the third frame of remote user 1, the third frame of remote user 2, and the third frame of remote user 3. Render the first target screen frame, the second target screen frame, and the third target screen frame in sequence according to the order of the preset processing times.

[0050] As described above, by obtaining multiple video streams, caching the latest video frames in each video stream, and at the preset processing time, merging the currently cached latest video frames of each video stream to obtain the target screen frame and rendering and displaying it. By adopting the above technical means, by caching the latest video frames in each video stream and only merging and outputting the currently cached latest video frames in each video stream, when the frame rate of a certain video stream is relatively high, it will not cause an increase in the data processing volume, thereby avoiding an increase in the data processing time, improving the working efficiency of the merged output of multiple video streams, and further reducing the stuttering situation of video calls and enhancing the user experience.

[0051] Based on the above embodiment, Figure 6 It is a schematic diagram of the structure of a video rendering device provided by an embodiment of the present application. Refer to Figure 6 , the video rendering device provided in this embodiment specifically includes: a video acquisition module 21 and a merging module 22.

[0052] Among them, the video acquisition module 21 is used to obtain multiple video streams, cache the latest video frames in each video stream, and obtain the currently cached video frames of each video stream;

[0053] The merging module 22 is used to, at the preset processing time, merge the currently cached latest video frames of each video stream to obtain the target screen frame, and render and display the target screen frame. The preset processing time is determined by the target frame rate.

[0054] In one embodiment, the video acquisition module 21 includes a caching unit;

[0055] A cache unit for obtaining multiple video streams. For each video stream, the newly obtained video frame is used to replace the originally cached video frame for caching, so as to obtain the currently cached video frame of each video stream.

[0056] In one embodiment, the video acquisition module 21 includes a video acquisition unit and a data processing unit;

[0057] The video acquisition unit is used to obtain multiple video streams;

[0058] The data processing unit is configured to, for each video stream, within a preset processing time period, if at least two video frames are received, perform frame dropping processing on the video frames to obtain the latest video frame of each video stream within the preset processing time period;

[0059] The cache unit is used to cache the latest video frame within the preset processing time period to obtain the currently cached video frame of each video stream.

[0060] In one embodiment, the video acquisition module 21 includes a timestamp determination unit;

[0061] The timestamp determination unit is configured to, for each video stream, within a preset processing time period, if at least two video frames are received, determine the latest video frame and the prior video frame within the preset processing time period according to the timestamps of the obtained video frames;

[0062] The data processing unit is used to delete the prior video frame and retain the latest video frame within the preset processing time period.

[0063] In one embodiment, the video acquisition unit is used to obtain multiple video streams;

[0064] The cache unit is configured to, for each video stream, within a preset processing time period, if one video frame is received, replace the originally cached video frame with the video frame for caching to obtain the currently cached video frame of the video stream.

[0065] In one embodiment, the video acquisition unit is used to obtain multiple video streams;

[0066] The cache unit is further configured to, for each video stream, within a preset processing time period, if no video frame is received, continue to cache the originally cached video frame to obtain the currently cached video frame of the video stream.

[0067] In one embodiment, the merging module 22 includes a merging unit and a rendering unit;

[0068] The merging unit is used to, at a preset processing time, merge the latest video frames currently cached for each video stream according to the preset distribution area of each video stream on the canvas to obtain a target picture frame corresponding to the preset processing time;

[0069] A rendering unit is configured to render and display a target video frame in sequence according to the sequence corresponding to the preset processing time.

[0070] As described above, by obtaining multiple video streams, caching the latest video frames in each video stream, and at the preset processing time, merging and processing the currently cached latest video frames of each video stream to obtain a target video frame for rendering and display. By adopting the above technical means, by caching the latest video frames in each video stream and only merging and outputting the currently cached latest video frames in each video stream, when the frame rate of a certain video stream is high, it will not cause an increase in the data processing volume, thereby avoiding an increase in the data processing time, improving the working efficiency of the merged output of multiple video streams, and further reducing the lag situation in video calls and enhancing the user experience.

[0071] The video rendering device provided by the embodiments of the present application can be used to execute the video rendering method provided by the above embodiments, and has corresponding functions and beneficial effects.

[0072] The embodiments of the present application provide a video rendering device. Referring to Figure 7 , the video rendering device includes: a processor 31, a memory 32, a communication module 33, an input device 34, and an output device 35. The number of processors in the video rendering device can be one or more, and the number of memories in the video rendering device can be one or more. The processor, memory, communication module, input device, and output device of the video rendering device can be connected through a bus or other means.

[0073] The memory 32, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as program instructions / modules corresponding to the video rendering method of any embodiment of the present application (for example, the video acquisition module and the merging module in the video rendering device). The memory mainly includes a program storage area and a data storage area. Among them, the program storage area can store an operating system and application programs required for at least one function; the data storage area can store data created according to the use of the device. In addition, the memory can include high-speed random access memory, and can also include non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some instances, the memory can further include a memory remotely provided with respect to the processor, and these remote memories can be connected to the device through a network. Examples of the above network include, but are not limited to, the Internet, an enterprise internal network, a local area network, a mobile communication network, and combinations thereof.

[0074] The communication module 33 is used for data transmission.

[0075] The processor 31 executes various functional applications and data processing of the device by running software programs, instructions, and modules stored in the memory, that is, implements the above video rendering method.

[0076] The input device 34 can be used to receive input digital or character information and generate key signal inputs related to the user settings and function controls of the device. The output device 35 can include display devices such as a display screen.

[0077] The above-provided video rendering device can be used to execute the video rendering method provided in the above embodiment, and has corresponding functions and beneficial effects.

[0078] The embodiment of the present application also provides a storage medium storing computer-executable instructions. The computer-executable instructions are used to execute a video rendering method when executed by a computer processor. The video rendering method includes: obtaining multiple video streams, caching the latest video frames in each video stream to obtain the currently cached video frames of each video stream; at a preset processing time, merging and processing the latest currently cached video frames of each video stream to obtain a target frame, and rendering and displaying the target frame. The preset processing time is determined by the target frame rate.

[0079] Storage medium - any of various types of memory devices or storage devices. The term "storage medium" is intended to include: installation media such as CD-ROMs, floppy disks, or magnetic tape devices; computer system memories or random access memories such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memories such as flash memories, magnetic media (such as hard disks or optical storage); registers or other similar types of memory elements, etc. The storage medium can also include other types of memories or combinations thereof. Additionally, the storage medium can be located in the first computer system in which the program is executed, or can be located in a different second computer system that is connected to the first computer system through a network (such as the Internet). The second computer system can provide program instructions to the first computer for execution. The term "storage medium" can include two or more storage media residing in different locations (such as in different computer systems connected through a network). The storage medium can store program instructions executable by one or more processors (such as specifically implemented as a computer program).

[0080] Certainly, the computer-executable instructions of the storage medium storing computer-executable instructions provided in the embodiment of the present application are not limited to the above video rendering method, and can also execute related operations in the video rendering methods provided in any embodiment of the present application.

[0081] The video rendering device, storage medium, and video rendering equipment provided in the above embodiments can execute the video rendering method provided in any embodiment of the present application. For technical details not described in detail in the above embodiments, reference can be made to the video rendering method provided in any embodiment of the present application.

[0082] The above is only the preferred embodiment of the present application and the technical principles applied. The present application is not limited to the specific embodiments here. Various obvious changes, re-adjustments, and substitutions that can be made by those skilled in the art will not depart from the protection scope of the present application. Therefore, although the present application has been described in more detail through the above embodiments, the present application is not limited to the above embodiments. Without departing from the concept of the present application, more other equivalent embodiments can be included, and the scope of the present application is determined by the scope of the claims.

Claims

1. A video rendering method, characterized in that, it includes: Obtain multiple video streams, cache the latest video frame in each video stream to obtain the currently cached video frame of each video stream; At a preset processing time, merge and process the latest video frames currently cached in each video stream to obtain a target picture frame, and render and display the target picture frame, where the preset processing time is determined by the target frame rate.

2. The method according to claim 1, characterized in that, The step of obtaining multiple video streams, caching the latest video frame in each video stream to obtain the currently cached video frame of each video stream includes: Obtain multiple video streams, and for each video stream, replace the originally cached video frame with the obtained latest video frame for caching to obtain the currently cached video frame of each video stream.

3. The method according to claim 1, characterized in that, The step of obtaining multiple video streams, caching the latest video frame in each video stream to obtain the currently cached video frame of each video stream includes: Obtain multiple video streams; For each video stream, within a preset processing time period, if at least two video frames are received, perform frame dropping processing on the video frames to obtain the latest video frame of each video stream within the preset processing time period; Cache the latest video frame within the preset processing time period to obtain the currently cached video frame of each video stream.

4. The method according to claim 3, characterized in that, The step of, for each video stream, within a preset processing time period, if at least two video frames are received, perform frame dropping processing on the video frames to obtain the latest video frame of each video stream within the preset processing time period includes: For each video stream, within a preset processing time period, if at least two video frames are received, determine the latest video frame and the prior video frame within the preset processing time period according to the timestamps of the obtained video frames; Delete the prior video frame and retain the latest video frame within the preset processing time period.

5. The method according to claim 1, characterized in that, The step of obtaining multiple video streams, caching the latest video frame in each video stream to obtain the currently cached video frame of each video stream includes: Obtain multiple video streams; For each video stream, within a preset processing time period, if one video frame is received, replace the originally cached video frame with the video frame for caching to obtain the currently cached video frame of the video stream.

6. The method according to claim 1, characterized in that, Before the step of, at a preset processing time, merging and processing the currently cached video frames of each video stream to obtain a target picture frame, it includes: Obtain multiple video streams; For each video stream, within a preset processing time period, if no video frame is received, continue to cache the originally cached video frame to obtain the currently cached video frame of the video stream.

7. The method according to claim 1, characterized in that, At the preset processing time, merging the currently cached video frames of each video stream to obtain a target picture frame, and rendering and displaying the target picture frame, includes: At the preset processing time, merging the latest video frames currently cached in each video stream according to the preset distribution area of each video stream in the canvas to obtain a target picture frame corresponding to the preset processing time; Rendering and displaying the target picture frames in sequence according to the sequence corresponding to the preset processing time.

8. A video rendering device Characterized in that It includes: A video acquisition module, configured to acquire multiple video streams, and cache the latest video frames in each video stream to obtain the currently cached video frames of each video stream; A merging module, configured to, at the preset processing time, merge the latest video frames currently cached in each video stream to obtain a target picture frame, and render and display the target picture frame, where the preset processing time is determined by the target frame rate.

9. A video rendering device Characterized in that It includes: A memory and one or more processors; The memory is used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1-7.

10. A storage medium storing computer-executable instructions Characterized in that The computer-executable instructions are used to execute the method according to any one of claims 1-7 when executed by a processor.

Images