Real-time video rendering method and device, and electronic equipment
By calculating the phase step of the video stream and performing block processing and interpolation calculations, the stuttering problem in video rendering under 5G networks was solved, achieving a stable and smooth video rendering effect.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-23
- Publication Date
- 2026-04-03
AI Technical Summary
Under 5G networks, traditional real-time video rendering methods are prone to image animation stuttering when frame rate or network fluctuations occur.
By acquiring real-time video streams, decoding to obtain multiple video frames, calculating the phase step size, and determining whether the total phase step size is an integer, if it is an integer, rendering is performed directly; otherwise, block processing and interpolation calculations are performed to stabilize the frame rate and avoid stuttering.
It avoids visual stuttering of image animations when the video frame rate fluctuates, thus improving the stability and smoothness of video rendering.
Smart Images
Figure CN121792784A_ABST
Abstract
Description
Technical Field
[0001] This application relates to electronic technology, and to, but is not limited to, a method, apparatus, and electronic device for rendering real-time video. Background Technology
[0002] With the development of the times and the progress of society, mobile 5G technology has gradually become popular. Under the 5G network, a variety of XR service capabilities based on video calls have been proposed, including dynamic image overlay, speech-to-text overlay, virtual background overlay, and face effect overlay. The core processing mechanism of these service capabilities includes real-time video stream encoding and decoding and rendering.
[0003] The traditional approach directly uses time alignment, which uses a selected timeline as the standard for multi-source data. Each data source aligns its own timeline with the standard timeline before rendering. However, when the frame rate or network fluctuates, the decrease in the frame rate of the video stream can cause visual stuttering in the image animation. Summary of the Invention
[0004] In view of this, embodiments of this application provide a method, apparatus, and electronic device for rendering real-time video.
[0005] The technical solution of this application embodiment is implemented as follows: This application provides a real-time video rendering method, the method comprising: acquiring a real-time video stream; decoding the real-time video stream to obtain a first frame video, a second frame video, and a third frame video; wherein the first frame video and the second frame video are adjacent, and the second frame video and the third frame video are adjacent; obtaining a first phase step size based on the first frame video and the second frame video; and obtaining a second phase step size based on the second frame video and the third frame video. Based on the first phase step and the second phase step, the total phase step is obtained; it is determined whether the total phase step is an integer, and a first determination result is obtained; if the first determination result is an integer, the real-time video stream is rendered using a first preset method.
[0006] Optionally, after determining whether the total phase step size is an integer and obtaining a first determination result, the process includes: if the first determination result is not an integer, obtaining a first region block based on the real-time video stream; determining whether the first region block satisfies a first preset condition and obtaining a second determination result; if the second determination result is yes, rendering the real-time video stream using the first preset method.
[0007] Optionally, after obtaining the second judgment result, the process includes: if the second judgment result is negative, determining whether the first region block satisfies the second preset condition, and obtaining the third judgment result.
[0008] Optionally, after obtaining the third judgment result, the process includes: if the third judgment result is yes, then a first image frame is obtained based on the real-time video stream, and the first region block is moved by 1 pixel value in the y-axis direction of the first image frame; if the third judgment result is no, then the first image frame is obtained based on the real-time video stream, and the first region block is moved by 1 pixel value in the x-axis direction of the first image frame.
[0009] Optionally, after obtaining the third judgment result, the process includes: determining whether the first region block meets the third preset condition to obtain a fourth judgment result; if the fourth judgment result is negative, then determining whether the first region block meets the fourth preset condition to obtain a fifth judgment result.
[0010] Optionally, after obtaining the fifth judgment result, the process includes: if the fifth judgment result is yes, then a second image frame is obtained based on the real-time video stream, and the first region block is moved by 1 pixel value in the y-axis direction of the second image frame; if the fifth judgment result is no, then a second image frame is obtained based on the real-time video stream, and the first region block is moved by 1 pixel value in the x-axis direction of the second image frame; wherein, the second image frame is adjacent to the first image frame.
[0011] Optionally, if the fifth judgment result is yes, the process includes: obtaining a second region block based on the real-time video stream; wherein the second region block is adjacent to the first region block; calculating the similarity between the first region block and the second region block to obtain a first similarity; and determining whether the first similarity is less than a first threshold to obtain a sixth judgment result.
[0012] Optionally, after obtaining the sixth judgment result, the process includes: if the sixth judgment result is yes, then obtaining the interpolation weight based on the total phase step size; and calculating the first hybrid result based on the interpolation weight using a first preset algorithm. Based on the first mixing result, determine whether the first region block meets the first preset condition.
[0013] Optionally, after obtaining the sixth judgment result, the process includes: if the sixth judgment result is negative, obtaining the first origin coordinates based on the first similarity; performing similarity calculation based on the first origin coordinates to obtain the first displacement vector; obtaining the interpolation weight based on the total phase step size; calculating the second mixing result based on the interpolation weight and the first displacement vector using a first preset algorithm; and determining whether the first region block meets the first preset condition based on the second mixing result.
[0014] Optionally, the real-time video stream is rendered using a first preset method. This includes: obtaining image rendering progress; determining whether the image rendering progress meets a fifth preset condition and obtaining a seventh judgment result; if the seventh judgment result is yes, then rendering the real-time video stream through a loop rendering mode or a freeze rendering mode.
[0015] Optionally, obtaining a first phase step based on the first video frame and the second video frame includes: obtaining a first interval time based on the first video frame and the second video frame; obtaining the image sequence frame rate and the video frame rate based on the real-time video stream; and obtaining the first phase step based on the first interval time, the image sequence frame rate, and the video frame rate.
[0016] A processing device includes: an acquisition unit, an analysis unit, and a processing unit; the acquisition unit is used to acquire a real-time video stream; the analysis unit is used to decode the real-time video stream to obtain a first frame video, a second frame video, and a third frame video; wherein the first frame video and the second frame video are adjacent, and the second frame video and the third frame video are adjacent; based on the first frame video and the second frame video, a first phase step size is obtained; based on the second frame video and the third frame video, a second phase step size is obtained; based on the first phase step size and the second phase step size, a total phase step size is obtained; and it is determined whether the total phase step size is an integer to obtain a first determination result; the processing unit is used to render the real-time video stream according to a first preset method if the first determination result is an integer.
[0017] Optionally, the processing unit includes: a first judgment module; the first judgment module is used to obtain a first region block based on the real-time video stream if the judgment result is not an integer after obtaining the judgment result.
[0018] Optionally, the processing unit includes: a second judgment module; the second judgment module is used to judge whether the first region block meets the first preset condition and obtain a second judgment result; if the second judgment result is yes, then the real-time video stream is rendered through the first preset method.
[0019] Optionally, the processing unit includes: a third judgment module; the third judgment module is used to determine whether the first region block meets the second preset condition if the second judgment result is negative, and to obtain a third judgment result.
[0020] Optionally, the processing unit includes: a first execution module; the first execution module is used to, after obtaining the third judgment result, if the third judgment result is yes, acquire a first image frame based on the real-time video stream, and move the first region block 1 pixel value in the y-axis direction of the first image frame.
[0021] Optionally, the processing unit includes: a second execution module; the second execution module is configured to, if the third judgment result is negative, acquire the first image frame based on the real-time video stream, and move the first region block by 1 pixel value in the x-axis direction of the first image frame.
[0022] Optionally, the processing unit includes a fourth judgment module; the fourth judgment module is used to determine whether the first region block meets the third preset condition after obtaining the third judgment result, and obtain the fourth judgment result.
[0023] Optionally, the processing unit includes a fifth judgment module; the fifth judgment module is used to determine whether the first region block meets the fourth preset condition if the fourth judgment result is negative, and to obtain the fifth judgment result.
[0024] Optionally, the processing unit includes: a third execution module; the third execution module is used to, after obtaining the fifth judgment result, if the fifth judgment result is yes, acquire a second image frame based on the real-time video stream, and the first region block moves 1 pixel value in the y-axis direction of the second image frame; wherein the second image frame is adjacent to the first image frame.
[0025] Optionally, the processing unit includes: a fourth execution module; the fourth execution module is used to acquire the second image frame based on the real-time video stream if the fifth judgment result is negative, and the first region block is moved by 1 pixel value in the x-axis direction of the second image frame.
[0026] Optionally, the processing unit includes: a sixth judgment module; the sixth judgment module is used to obtain a second region block based on the real-time video stream; wherein the second region block is adjacent to the first region block; calculate the similarity between the first region block and the second region block to obtain a first similarity; and determine whether the first similarity is less than a first threshold to obtain a sixth judgment result.
[0027] Optionally, the processing unit includes: a first calculation module; the first calculation module is configured to, if the sixth judgment result is yes, obtain interpolation weights based on the total phase step size; calculate based on the interpolation weights using a first preset algorithm to obtain a first mixing result; and determine whether the first region block satisfies a first preset condition based on the first mixing result.
[0028] Optionally, the processing unit includes: a second calculation module; the second calculation module is configured to, if the sixth judgment result is negative, obtain the first origin coordinates based on the first similarity; perform similarity calculation based on the first origin coordinates to obtain a first displacement vector; obtain interpolation weights based on the total phase step size; calculate a second mixing result based on the interpolation weights and the first displacement vector using a first preset algorithm; and determine whether the first region block satisfies the first preset condition based on the second mixing result.
[0029] Optionally, the processing unit includes: a seventh judgment module; the seventh judgment module is used to obtain the image rendering progress; determine whether the image rendering progress meets the fifth preset condition, and obtain the seventh judgment result; if the seventh judgment result is yes, then render the real-time video stream through a loop rendering mode or a freeze rendering mode.
[0030] Optionally, the analysis unit includes: a phase module; the phase module is configured to obtain a first interval time based on the first video frame and the second video frame; obtain the image sequence frame rate and the video frame rate based on the real-time video stream; and obtain the first phase step size based on the first interval time, the image sequence frame rate, and the video frame rate.
[0031] An electronic device includes: a memory for storing at least one set of instructions; a processor for acquiring a real-time video stream; decoding the real-time video stream to obtain a first frame, a second frame, and a third frame; wherein the first frame and the second frame are adjacent, and the second frame and the third frame are adjacent; obtaining a first phase step based on the first frame and the second frame; obtaining a second phase step based on the second frame and the third frame; obtaining a total phase step based on the first phase step and the second phase step; determining whether the total phase step is an integer to obtain a first determination result; if the first determination result is an integer; and rendering the real-time video stream using a first preset method.
[0032] This invention provides a real-time video rendering method, apparatus, and electronic device. First, a real-time video stream is acquired, then the real-time video stream is decoded to obtain a first frame, a second frame, and a third frame. The first and second frames are adjacent, as are the second and third frames. Based on the first and second frames, a first phase step size is obtained; based on the second and third frames, a second phase step size is obtained. The first and second phase step sizes are then added to obtain a total phase step size. It is then determined whether the total phase step size is an integer. If it is, the real-time video stream is rendered using a first preset method. In other words, this solution renders the real-time video stream based on the total phase step size, thereby avoiding visual stuttering in image animation. Attached Figure Description
[0033] Figure 1 A flowchart illustrating a real-time video rendering method provided in an embodiment of the present invention; Figure 2 Another flowchart of the real-time video rendering method provided in the embodiments of the present invention; Figure 3 Another flowchart of the real-time video rendering method provided in the embodiments of the present invention; Figure 4 Another flowchart of the real-time video rendering method provided in the embodiments of the present invention; Figure 5 Another flowchart of the real-time video rendering method provided in the embodiments of the present invention; Figure 6 Another flowchart of the real-time video rendering method provided in the embodiments of the present invention; Figure 7 Another flowchart of the real-time video rendering method provided in the embodiments of the present invention; Figure 8 Another flowchart of the real-time video rendering method provided in the embodiments of the present invention; Figure 9 Another flowchart of the real-time video rendering method provided in the embodiments of the present invention; Figure 10 Another flowchart of the real-time video rendering method provided in the embodiments of the present invention; Figure 11 A schematic diagram of a rendering apparatus provided in an embodiment of the present invention; Figure 12 This is a schematic diagram of the structural composition of an electronic device provided in an embodiment of the present invention. Detailed Implementation
[0034] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0035] Please refer to Figure 1 ,in, Figure 1 A flowchart illustrating an implementation of the real-time video rendering method provided in this application embodiment may include: Step S101: Acquire real-time video stream; Step S102: Decode the real-time video stream to obtain the first frame, the second frame, and the third frame; wherein the first frame and the second frame are adjacent, and the second frame and the third frame are adjacent. Step S103: Obtain the first phase step size based on the first and second video frames; Step S104: Obtain the second phase step size based on the second and third video frames; Step S105: Obtain the total phase step based on the first phase step and the second phase step; Step S106: Determine whether the total phase step size is an integer, and obtain the first determination result; Step S107: If the first judgment result is an integer, then render the real-time video stream using the first preset method.
[0036] The acquired real-time video stream is the original video to be rendered. This original video is then decoded to obtain multiple video frames, including at least a first frame, a second frame, and a third frame. The first and second frames are adjacent, and the second and third frames are adjacent. Specifically, the phase step size between adjacent frames is obtained, and the phase step sizes of each frame are summed to obtain the total phase step size. If the total phase step size is an integer, rendering is performed using a first preset method. This first preset method can be to extract the corresponding image frame from the image sequence according to its index value, input the video frame and the image frame into the rendering engine for overlay, and use common image overlay algorithms, such as linear weighted blending or alpha blending, to output the image frame. It is superimposed onto the corresponding video frame.
[0037] The real-time video stream may also include: the fourth frame, the fifth frame, the sixth frame, and so on.
[0038] Specifically, the first step is to determine the rendering frame rate of the image sequence from the source animation from which the image sequence needs to be overlaid. High-precision clocks are used to record the time it takes for each frame of the video stream to reach its destination. (Unit: seconds) Calculate the time interval between the current frame and the previous frame using the following formula: .in This represents the time value recorded in the previous frame of the video stream. Continuous calculation and dynamic adjustment are necessary to achieve strict time synchronization. The dynamic phase step size can be obtained after obtaining the time interval, calculated using the following formula: .in Used to indicate the rendering progress of the image sequence corresponding to each video frame, that is, the number of images that should be rendered within each video frame time. frame, This is the current video frame number. It's important to note that when the instantaneous frame rate is too low or infinitely close to 0, This can approach infinity, potentially causing phase value overflow or causing the index to jump directly back to the starting frame after the modulo operation, disrupting the continuity of rendering. A frame rate lower limit is set; when the frame rate is detected to be below the set threshold, subsequent steps are forcibly paused. Simultaneously, an upper limit is set for the dynamic step size (usually no greater than the image sequence length) to prevent jumps or index overflows caused by extreme frame rates.
[0039] Step 2: Initialize the phase The phase is accumulated once for each frame of the video stream processed, and the calculation formula is as follows: ,in The phase of the previous image frame indexed by the current image. Increment the sequence number of the current video frame. Using Formula 1: And Formula 2: This yields the image sequence index and image compensation interpolation weights corresponding to the current video frame. Dynamic phase accumulation requires that the total phase accumulation value be strictly equal to the image sequence length. The calculation formula is: ,in For the total time, The total duration of the entire image sequence overlay is expressed by the formula. Therefore, the dynamic phase accumulator will automatically compensate for the time error caused by frame rate fluctuations.
[0040] Specifically, the advantages of the video stream synchronous rendering method based on dynamic phase accumulator also include: obtaining the timestamp difference between adjacent frames of the real-time video stream. and image sequence source animation frame rate The phase step size calculation formula is used. The phase accumulator is dynamically updated to eliminate accumulated errors caused by fluctuations in the video stream frame rate and to calculate the phase of each frame. The phase value is divided into integer parts. and decimal part The integer part indexes the first image in the image sequence. The frame is used as the frame to be rendered, while the fractional part is used as interpolation weights in the calculation of blending adjacent image frames. The fractional part preservation mechanism avoids the frame jump problem caused by traditional integer indexing methods, achieving sub-frame level synchronization accuracy between video streams and image sequences.
[0041] Step 3: If the judgment result is not an integer, proceed as follows: Perform block search in the current image frame and the next image frame, and use the absolute difference and SAD matching similarity to obtain the optimal motion vector for each block. Use interpolation weights and the optimal motion vector to calculate the RGB components of the mixed pixels in the blocks. After completing the block traversal of the current image frame, the output image frame is obtained.
[0042] Step 31: Determine whether the phase in Step 2 is an integer. If it is, there is no need to perform block matching and compensation interpolation calculations. Directly use the image frame corresponding to the frame index as the rendering frame for Step 4; otherwise, continue to Step 32.
[0043] Please refer to Figure 2 The method in this embodiment may include: determining whether the total phase step size is an integer, and after obtaining a first determination result, including: Step S201: If the first judgment result is not an integer, then obtain the first region block based on the real-time video stream; Step S202: Determine whether the first region block meets the first preset condition and obtain the second determination result; Step S203: If the second judgment result is yes, then render the real-time video stream using the first preset method.
[0044] The first region block is according to Size selects the current image frame The rectangular area is used as a block Whether the first region block meets the first preset condition is the judgment block. Does it meet the requirements? .
[0045] Specifically, step 32: Record the current image frame. The width is Gao Wei Record a frame of an image. The width is , Gao Wei The unit is pixels (px). Typically, each frame in an image sequence has the same size, therefore it can be assumed that... , The unit is pixels (px). From the top left corner... Begin, according to Size selects the current image frame The rectangular area is used as a block ,remember In image frame pixel coordinates Let the origin be denoted as . In image frame pixel coordinates For punctuation, initially .make From image frame of The position traversal begins, and the coordinates of the punctuation point are now displayed. Turn to step 34.
[0046] Specifically, step 33, judgment block Does it meet the requirements? If so, it is considered to be for image frames. After the block traversal is complete, proceed to step four, which involves using common image overlay algorithms, such as linear weighted blending or alpha blending, to output the image frames. It is superimposed onto the corresponding video frame.
[0047] Please refer to Figure 3 The method in this embodiment may include: after obtaining the second judgment result, including: Step S301: If the second judgment result is negative, then determine whether the first region block meets the second preset condition and obtain the third judgment result; Specifically, determining whether the first region block satisfies the second preset condition involves determining whether the first region block satisfies... In other words, after obtaining the total phase step size, if the phase step size is not an integer but a decimal, the phase step size is judged a second time. The rendering is then performed based on the result of the two judgments, thereby avoiding visual stuttering of the image animation.
[0048] After obtaining the third judgment result, it includes: Step S302: If the third judgment result is yes, then the first image frame is obtained based on the real-time video stream, and the first region block is moved by 1 pixel value in the y-axis direction of the first image frame; Step S303: If the third judgment result is negative, then the first image frame is obtained based on the real-time video stream, and the first region block is moved by 1 pixel value in the x-axis direction of the first image frame.
[0049] Specifically, in step 33, first determine the block. Does it meet the requirements? If so, it is considered to be for image frames. If the block traversal is complete, proceed to step four; otherwise, check the block... Does it meet the requirements? If so, make the block To image frame Moving 1 pixel along the y-axis changes the origin coordinates to Otherwise, make the block To image frame Moving 1 pixel along the x-axis changes the origin coordinates to... .make From image frame of The position traversal begins; proceed to step 35.
[0050] Please refer to Figure 4 The method in this embodiment may include: after obtaining the third judgment result, including: Step S401: Determine whether the first region block meets the third preset condition and obtain the fourth judgment result; Step S402: If the fourth judgment result is negative, then determine whether the first region block meets the fourth preset condition and obtain the fifth judgment result; Step S403: If the fifth judgment result is yes, then the second image frame is obtained based on the real-time video stream, and the first region block is moved by 1 pixel value in the y-axis direction of the second image frame; Step S404: If the fifth judgment result is negative, then the second image frame is obtained based on the real-time video stream, and the first region block is moved by 1 pixel value in the x-axis direction of the second image frame.
[0051] Specifically, determining whether the first region block satisfies the third preset condition is whether the first region block satisfies... ; Determine whether the first region block satisfies the fourth preset condition. .
[0052] Specifically, step 34, the judgment block. Does it meet the requirements? If so, it is considered to be for image frames. If the block matching ends, proceed to step 36; otherwise, determine the block. Does it meet the requirements? If so, make the block To image frame Moving the y-axis by 1 pixel changes the punctuation coordinates as follows: Otherwise, make the block To image frame Moving the pointer 1 pixel along the x-axis changes the pointer's coordinates as follows: Continue to step 35.
[0053] Please refer to Figure 5 The method in this embodiment may include: if the fifth determination result is yes, including: Step S501: Obtain a second region block based on the real-time video stream; wherein the second region block is adjacent to the first region block; Step S502: Calculate the similarity between the first region block and the second region block to obtain the first similarity score; Step S503: Determine whether the first similarity is less than the first threshold, and obtain the sixth judgment result.
[0054] Specifically, step 35, using blocks The punctuation coordinates in the image frame According to Take out the block of size Using absolute difference and calculation blocks and blocks The similarity is calculated using the formula: . Result and block The punctuation coordinates are stored as key-value pairs in a hash table. In this context, a similarity score closer to 0 indicates a higher similarity. This is used to determine whether the condition is met. If so, it is considered to be already in the image frame. Find the block that matches the maximum similarity If the traversal is terminated, proceed to step 36; otherwise, proceed to step 34.
[0055] Step 36, Traverse Get the maximum similarity ,judge Is it less than the set threshold? If yes, proceed to step 37; otherwise, proceed to step 38.
[0056] Please refer to Figure 6 The method in this embodiment may include: after obtaining the sixth judgment result, including: Step S601: If the result of the sixth judgment is yes, then the interpolation weight is obtained based on the total phase step size; Step S602: Calculate the first hybrid result based on the interpolation weights using the first preset algorithm; Step S603: Determine whether the first region block meets the first preset condition based on the first mixing result.
[0057] The first preset algorithm can be a traditional linear interpolation algorithm.
[0058] Specifically, in step 37, when At that time, based on the interpolation weights obtained in step two The downgrade uses a traditional linear interpolation algorithm to iterate through and calculate the current image frame. and the next image frame The blending result of pixel values is calculated using the following formula: ,in The coordinates of the pixels are traversed, and the output image frame is obtained after a complete traversal and calculation. Proceed to step 33.
[0059] Please refer to Figure 7 The method in this embodiment may include: after obtaining the sixth judgment result, including: Step S701: If the result of the sixth judgment is negative, then obtain the coordinates of the first origin based on the first similarity. Step S702: Perform similarity calculation based on the coordinates of the first origin to obtain the first displacement vector; Step S703: Obtain the interpolation weights based on the total phase step size; Step S704: Based on the interpolation weights and the first displacement vector, the second hybrid result is obtained by calculating using the first preset algorithm. Step S705: Determine whether the first region block meets the first preset condition based on the second mixing result.
[0060] Specifically, step 38: Based on the maximum similarity Find the origin coordinates and calculate the coordinates in the image frame. Middle and Block Most similar blocks The displacement vector between them is calculated using the following formula: Continue to step 39; Step 39: Based on the interpolation weights obtained in Step 2 traverse and calculate the current image frame and the next image frame The blending result of pixel values is calculated using the following formula: ,in The coordinates of the pixels are traversed, and the output image frame is obtained after a complete traversal and calculation. Proceed to step 33.
[0061] Specifically, pixel-level motion compensation is achieved by acquiring pixel blocks of a specific size in the current image frame and searching block by block in the next image frame. An absolute difference method is used for simple and efficient similarity measurement. The similarity between blocks is calculated within the search window, and the displacement vector of the pixel block between two frames is tracked based on the maximum similarity. Motion compensation interpolation is performed based on interpolation weights and the displacement vector, solving the problem of motion blur or frame jumps in complex texture areas caused by frame transitions in video rendering scenarios. When the maximum similarity is below a set threshold, it can degenerate into linear interpolation. This dynamic switching interpolation algorithm solves the problem of artifacts generated in moving areas by traditional linear interpolation while ensuring interpolation stability in low-texture areas.
[0062] Please refer to Figure 8 The method in this embodiment may include: rendering real-time video using a first preset method. Frequency streams, including: Step S801: Obtain image rendering progress; Step S802: Determine whether the image rendering progress meets the fifth preset condition and obtain the seventh judgment result; Step S803: If the result of the seventh judgment is yes, then render the real-time video stream using either loop rendering mode or freeze rendering mode.
[0063] Specifically, in step four, common image overlay algorithms, such as linear weighted blending or alpha blending, are used to output image frames. Superimposed onto the corresponding video frame; Step 5: Determine if the image rendering progress meets the requirements of Step 2. ,in This is the index of the image sequence corresponding to the current video frame. This represents the length of the image sequence. If so, the rendering mode configured by the user can be categorized as follows: 1. Loop rendering mode: The image sequence is cyclically superimposed onto the video stream to achieve a continuously playing animation effect. The calculation formula is... ,in express 1. Take the modulo of image sequence L to modify the index to the start position of the sequence; 2. Freeze rendering mode: After the image sequence loops once, the last frame is rendered forever to achieve a freeze effect. The calculation formula is as follows: Otherwise, continue with the rendered video encoding or other steps.
[0064] Please refer to Figure 9 The method in this embodiment may include: obtaining a first phase step size based on a first frame of video and a second frame of video, including: Step S901: Obtain the first interval time based on the first frame and the second frame of video; Step S902: Based on the real-time video stream, obtain the image sequence frame rate and video frame rate; Step S903: Obtain the first phase step size based on the first interval time, the image sequence frame rate, and the video frame rate.
[0065] Specifically, based on the time interval between two frames, the frame rate of the image sequence, and the frame rate of the video, the first phase step is calculated. When processing the video stream, the phase step is accumulated for each frame. That is, the first phase step, the second phase step, the third phase step, and the Nth phase step are added together to obtain the total phase step. The processing and rendering are then performed based on the total phase step.
[0066] Please refer to Figure 10The method in this embodiment may include: taking the above figure as an example, after receiving the raw input stream pushed by the media server MS, the XR engine decodes the video into video frames and inputs the decoding completion time to the dynamic phase accumulator. The accumulator calculates the step size based on the time interval between two frames, and then calculates the phase based on the step size. If the calculated phase is an integer, the corresponding image frame is retrieved from the image sequence according to the index value, and the video frame and the image frame are input to the rendering engine for superposition. If the phase has a decimal value, the decimal value is used as the interpolation weight to generate an index and a mixed frame of the next frame of the index as the frame to be rendered and input to the rendering engine for superposition. The rendering engine outputs the superposition result to the XR engine, which encodes and outputs it.
[0067] Please refer to Figure 11 The apparatus of this embodiment may include the following structure: Acquisition unit 1001 is used to acquire real-time video streams; Analysis unit 1002 is used to decode a real-time video stream to obtain a first frame, a second frame, and a third frame; wherein the first frame and the second frame are adjacent, and the second frame and the third frame are adjacent; based on the first and second frames, a first phase step size is obtained; based on the second and third frames, a second phase step size is obtained; based on the first and second phase step sizes, a total phase step size is obtained; and it is determined whether the total phase step size is an integer to obtain a first determination result. The processing unit 1003 is used to render a real-time video stream in a first preset manner if the first judgment result is an integer.
[0068] The method in this embodiment may include: the processing unit includes: a first judgment module; the first judgment module is used to obtain a judgment result, and if the judgment result is not an integer, then obtain a first region block based on the real-time video stream.
[0069] The method of this embodiment may include: the processing unit includes: a second judgment module; the second judgment module is used to judge whether the first region block meets the first preset condition and obtain a second judgment result; if the second judgment result is yes, then the real-time video stream is rendered through the first preset method.
[0070] The method in this embodiment may include: the processing unit includes: a third judgment module; the third judgment module is used to determine whether the first region block meets the second preset condition if the second judgment result is negative, and to obtain the third judgment result.
[0071] The method of this embodiment may include: the processing unit includes: a first execution module; the first execution module is used to obtain a third judgment result, and if the third judgment result is yes, then obtain a first image frame based on the real-time video stream, and the first region block moves 1 pixel value in the y-axis direction of the first image frame.
[0072] The method of this embodiment may include: the processing unit includes: a second execution module; the second execution module is used to acquire a first image frame based on the real-time video stream if the third judgment result is negative, and the first region block moves 1 pixel value in the x-axis direction of the first image frame.
[0073] The method in this embodiment may include: the processing unit includes: a fourth judgment module; the fourth judgment module is used to determine whether the first region block meets the third preset condition after obtaining the third judgment result, and obtain the fourth judgment result.
[0074] The method in this embodiment may include: the processing unit includes: a fifth judgment module; the fifth judgment module is used to determine whether the first region block meets the fourth preset condition if the fourth judgment result is negative, and to obtain the fifth judgment result.
[0075] The method of this embodiment may include: the processing unit includes: a third execution module; the third execution module is used to obtain a fifth judgment result, and if the fifth judgment result is yes, then obtain a second image frame based on the real-time video stream, and the first region block moves 1 pixel value in the y-axis direction of the second image frame; wherein, the second image frame is adjacent to the first image frame.
[0076] The method of this embodiment may include: the processing unit includes: a fourth execution module; the fourth execution module is used to acquire a second image frame based on the real-time video stream if the fifth judgment result is negative, and the first region block moves 1 pixel value in the x-axis direction of the second image frame.
[0077] The method of this embodiment may include: the processing unit includes: a sixth judgment module; the sixth judgment module is used to obtain a second region block based on a real-time video stream; wherein the second region block is adjacent to a first region block; calculate the similarity between the first region block and the second region block to obtain a first similarity; and determine whether the first similarity is less than a first threshold to obtain a sixth judgment result.
[0078] The method of this embodiment may include: the processing unit includes: a first calculation module; the first calculation module is used to obtain interpolation weights based on the total phase step size if the sixth judgment result is yes; to calculate the first mixing result based on the interpolation weights using a first preset algorithm; and to determine whether the first region block meets the first preset condition based on the first mixing result.
[0079] The method of this embodiment may include: the processing unit includes: a second calculation module; the second calculation module is used to obtain the coordinates of the first origin based on the first similarity if the sixth judgment result is negative; to perform similarity calculation based on the coordinates of the first origin to obtain the first displacement vector; to obtain the interpolation weight based on the total phase step size; to calculate the second mixing result based on the interpolation weight and the first displacement vector through a first preset algorithm; and to determine whether the first region block meets the first preset condition based on the second mixing result.
[0080] The method of this embodiment may include: the processing unit includes: a seventh judgment module; the seventh judgment module is used to obtain the image rendering progress; to determine whether the image rendering progress meets the fifth preset condition, and to obtain the seventh judgment result; if the seventh judgment result is yes, then the real-time video stream is rendered through the loop rendering mode or the freeze rendering mode.
[0081] The method in this embodiment may include: the analysis unit includes: a phase module; the phase module is used to obtain a first interval time based on a first frame video and a second frame video; obtain an image sequence frame rate and a video frame rate based on a real-time video stream; and obtain a first phase step size based on the first interval time, the image sequence frame rate, and the video frame rate.
[0082] Please refer to Figure 12 This embodiment of the present application also discloses an electronic device, which includes at least one processor 701, and at least one memory 702 and a bus 703 connected to the processor 701; wherein the processor 701 and the memory 702 communicate with each other through the bus 703; the processor 701 is used to call program instructions in the memory 702 to execute the above-mentioned real-time video rendering method.
[0083] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A method for rendering real-time video, characterized in that, The method includes: Acquire real-time video stream; Decode the real-time video stream to obtain a first frame, a second frame, and a third frame; wherein the first frame and the second frame are adjacent, and the second frame and the third frame are adjacent. Based on the first frame of video and the second frame of video, the first phase step size is obtained; Based on the second and third video frames, a second phase step size is obtained; The total phase step size is obtained based on the first phase step size and the second phase step size; Determine whether the total phase step size is an integer to obtain a first determination result; If the first determination result is an integer, then the real-time video stream is rendered using the first preset method.
2. The method according to claim 1, characterized in that, After determining whether the total phase step size is an integer and obtaining the first determination result, the process includes: If the first judgment result is not an integer, then a first region block is obtained based on the real-time video stream; Determine whether the first region block meets the first preset condition, and obtain the second determination result; If the second determination result is yes, then the real-time video stream is rendered using the first preset method.
3. The method according to claim 2, characterized in that, After obtaining the second judgment result, the following is included: If the second judgment result is negative, then determine whether the first region block meets the second preset condition and obtain the third judgment result.
4. The method according to claim 3, characterized in that, After obtaining the third judgment result, the following is included: If the third judgment result is yes, then the first image frame is obtained based on the real-time video stream, and the first region block is moved by 1 pixel value in the y-axis direction of the first image frame; If the third judgment result is negative, then the first image frame is obtained based on the real-time video stream, and the first region block is moved by 1 pixel value in the x-axis direction of the first image frame.
5. The method according to claim 4, characterized in that, After obtaining the third judgment result, the following is included: Determine whether the first region block meets the third preset condition, and obtain the fourth determination result; If the fourth judgment result is negative, then it is determined whether the first region block meets the fourth preset condition, and a fifth judgment result is obtained.
6. The method according to claim 5, characterized in that, After obtaining the fifth judgment result, the following is included: If the fifth judgment result is yes, then the second image frame is obtained based on the real-time video stream, and the first region block is moved by 1 pixel value in the y-axis direction of the second image frame; If the fifth judgment result is negative, then the second image frame is obtained based on the real-time video stream, and the first region block is moved by 1 pixel value in the x-axis direction of the second image frame; The second image frame is adjacent to the first image frame.
7. The method according to claim 5, characterized in that, If the fifth judgment result is yes, it includes: Based on the real-time video stream, a second region block is obtained; wherein the second region block is adjacent to the first region block; Calculate the similarity between the first region block and the second region block to obtain the first similarity. Determine whether the first similarity is less than the first threshold to obtain the sixth determination result.
8. The method according to claim 7, characterized in that, After obtaining the sixth judgment result, the following is included: If the sixth judgment result is yes, then the interpolation weight is obtained based on the total phase step size; The first hybrid result is obtained by calculating based on the interpolation weights using a first preset algorithm; Based on the first mixing result, determine whether the first region block meets the first preset condition.
9. The method according to claim 7, characterized in that, After obtaining the sixth judgment result, the following is included: If the sixth judgment result is negative, then the coordinates of the first origin are obtained based on the first similarity. A similarity calculation is performed based on the coordinates of the first origin to obtain the first displacement vector; Based on the total phase step size, the interpolation weights are obtained; Based on the interpolation weights and the first displacement vector, a second hybrid result is obtained by calculating using a first preset algorithm. Based on the second mixing result, determine whether the first region block meets the first preset condition.
10. The method according to claim 1, characterized in that, The real-time view is rendered using a first preset method. Frequency streams, including: Get image rendering progress; Determine whether the image rendering progress meets the fifth preset condition, and obtain the seventh determination result; If the seventh judgment result is yes, then the real-time video stream is rendered using either loop rendering mode or freeze rendering mode.
11. The method according to claim 1, characterized in that, Based on the first frame of video and the second frame of video, Obtaining the first phase step includes: The first interval time is obtained based on the first frame of video and the second frame of video; Based on the real-time video stream, the image sequence frame rate and video frame rate are obtained; The first phase step size is obtained based on the first interval time, the image sequence frame rate, and the video frame rate.
12. A rendering apparatus, characterized in that, The device includes: an acquisition unit, an analysis unit, and a processing unit. The acquisition unit is used to acquire real-time video streams; The analysis unit is used to decode the real-time video stream to obtain a first frame, a second frame, and a third frame; wherein the first frame and the second frame are adjacent, and the second frame and the third frame are adjacent; based on the first frame and the second frame, a first phase step size is obtained; based on the second frame and the third frame, a second phase step size is obtained; based on the first phase step size and the second phase step size, a total phase step size is obtained; and it is determined whether the total phase step size is an integer to obtain a first determination result. The processing unit is configured to render the real-time video stream in a first preset manner if the first determination result is an integer.
13. The method according to claim 12, characterized in that, The processing unit includes: a first judgment module; The first judgment module is used to obtain a first region block based on the real-time video stream if the judgment result is not an integer after obtaining the judgment result.
14. The method according to claim 13, characterized in that, The processing unit includes: a second judgment module; The second judgment module is used to determine whether the first region block meets the first preset condition and obtain a second judgment result; if the second judgment result is yes, the real-time video stream is rendered through the first preset method.
15. The method according to claim 14, characterized in that, The processing unit includes: a third judgment module; The third judgment module is used to determine whether the first region block meets the second preset condition if the second judgment result is negative, and to obtain the third judgment result.
16. The method according to claim 15, characterized in that, The processing unit includes: a first execution module; The first execution module is configured to, after obtaining the third judgment result, if the third judgment result is yes, acquire the first image frame based on the real-time video stream, and move the first region block by 1 pixel value in the y-axis direction of the first image frame.
17. The method according to claim 16, characterized in that, The processing unit includes: a second execution module; The second execution module is configured to, if the third judgment result is negative, acquire the first image frame based on the real-time video stream, and move the first region block by 1 pixel value in the x-axis direction of the first image frame.
18. The method according to claim 17, characterized in that, The processing unit includes: a fourth judgment module; The fourth judgment module is used to determine whether the first region block meets the third preset condition after obtaining the third judgment result, and then obtain the fourth judgment result.
19. The method according to claim 18, characterized in that, The processing unit includes: a fifth judgment module; The fifth judgment module is used to determine whether the first region block meets the fourth preset condition if the fourth judgment result is negative, and to obtain the fifth judgment result.
20. The method according to claim 19, characterized in that, The processing unit includes: a third execution module; The third execution module is used to, after obtaining the fifth judgment result, if the fifth judgment result is yes, acquire the second image frame based on the real-time video stream, and move the first region block by 1 pixel value in the y-axis direction of the second image frame; wherein the second image frame is adjacent to the first image frame.
21. The method according to claim 20, characterized in that, The processing unit includes: a fourth execution module; The fourth execution module is used to obtain the second image frame based on the real-time video stream if the fifth judgment result is negative, and the first region block is moved by 1 pixel value in the x-axis direction of the second image frame.
22. The method according to claim 21, characterized in that, The processing unit includes: a sixth judgment module; The sixth judgment module is used to obtain a second region block based on the real-time video stream; wherein the second region block is adjacent to the first region block; calculate the similarity between the first region block and the second region block to obtain a first similarity; and determine whether the first similarity is less than a first threshold to obtain a sixth judgment result.
23. The method according to claim 22, characterized in that, The processing unit includes: a first computing module; The first calculation module is configured to, if the sixth judgment result is yes, obtain interpolation weights based on the total phase step size; calculate the first mixing result based on the interpolation weights using a first preset algorithm; and determine whether the first region block satisfies the first preset condition based on the first mixing result.
24. The method according to claim 23, characterized in that, The processing unit includes: a second computing module; The second calculation module is used to, if the sixth judgment result is negative, obtain the first origin coordinates based on the first similarity; perform similarity calculation based on the first origin coordinates to obtain the first displacement vector; obtain the interpolation weight based on the total phase step; calculate the second mixing result based on the interpolation weight and the first displacement vector using a first preset algorithm; and determine whether the first region block meets the first preset condition based on the second mixing result.
25. The method according to claim 24, characterized in that, The processing unit includes: a seventh judgment module; The seventh judgment module is used to obtain the image rendering progress; determine whether the image rendering progress meets the fifth preset condition, and obtain the seventh judgment result; if the seventh judgment result is yes, then render the real-time video stream through loop rendering mode or freeze rendering mode.
26. The method according to claim 25, characterized in that, The analysis unit includes: a phase module; The phase module is configured to obtain a first interval time based on the first video frame and the second video frame; obtain the image sequence frame rate and the video frame rate based on the real-time video stream; and obtain the first phase step size based on the first interval time, the image sequence frame rate, and the video frame rate.
27. An electronic device, characterized in that, include: Memory, used to store at least one set of instructions; Processor used to acquire real-time video streams; Decode the real-time video stream to obtain a first frame, a second frame, and a third frame; wherein the first frame and the second frame are adjacent, and the second frame and the third frame are adjacent. Based on the first frame of video and the second frame of video, the first phase step size is obtained; Based on the second and third video frames, a second phase step size is obtained; The total phase step size is obtained based on the first phase step size and the second phase step size; Determine whether the total phase step size is an integer to obtain a first determination result; If the first determination result is an integer, then the real-time video stream is rendered using the first preset method.