Real-time rendering method and processing method, system, device and storage medium of video

Abstract

This application provides a real-time video rendering method, processing method, system, apparatus, and storage medium. The real-time video rendering method includes: loading a first video; for any first video frame in the first video, determining a rendering perspective; rendering a second video frame based on the first video frame and the rendering perspective; and adjusting the rendering perspective in response to a perspective control event, using it as the rendering perspective for the next first video frame. This method can process wide-angle videos in real time based on preset processing logic or user-specified logic, ultimately obtaining a processed video stream. It effectively solves the problems of the inability to analyze and render wide-angle videos in real time and low user interactivity in existing technologies. It not only significantly expands the applicable scenarios of wide-angle video rendering methods and improves scene adaptability and processing timeliness, but also enhances the flexibility of wide-angle video rendering output perspective and the utilization rate of video data resources according to the user's personalized needs.

Description

Technical Field

[0001] The embodiments of this application relate to the field of video processing technology, and in particular to a real-time video rendering method, system, apparatus and storage medium. Background Technology

[0002] Existing wide-angle video processing methods mostly employ tracking algorithms to analyze and crop the captured footage. For example, CN111163267A discloses a panoramic video editing method, apparatus, device, and storage medium. This method detects salient targets in panoramic video frames, tracks the detected salient targets using a preset target tracking algorithm, and edits the panoramic video based on the forward direction viewpoint and the viewpoint where the salient target is located, generating a target video corresponding to the panoramic video. It is evident that existing technologies typically perform one-time editing of wide-angle videos based on preset logic. However, when facing situations such as live sports events requiring real-time footage capture and frame-by-frame analysis of the output viewpoint, or when users need to manually control the output viewpoint, such methods have significant shortcomings.

[0003] First, existing technologies largely focus on processing existing wide-angle videos, lacking effective methods for processing real-time wide-angle video footage. In scenarios such as live sports broadcasts, large-scale cultural and sports events, and public safety monitoring, it is difficult to perform real-time perspective analysis and adjustment based on the currently captured footage and output rendered images, thus failing to create a complete and continuous processed video stream. Therefore, existing technologies are limited by post-processing of wide-angle videos, resulting in limitations in their application scenarios and processing time, and failing to meet the needs of real-time shooting scenarios.

[0004] Secondly, existing technologies typically process wide-angle videos in a one-time manner based on preset logic. That is, they use a fixed algorithm provided by the technical solution to process the wide-angle video stream using a pre-defined tracking and processing logic. Ultimately, they can only output a processed video stream with the same viewpoint change process, failing to enable customized viewpoint output and user interaction—that is, allowing users to control and change their viewing angle, and processing the wide-angle video stream based on user-specified viewpoint changes. This results in a rigid wide-angle video rendering perspective, making it difficult to meet users' personalized output needs, and leading to low utilization of video data resources. Summary of the Invention

[0005] In view of this, this application provides a method, system and apparatus for real-time rendering of wide-angle video based on viewpoint control events and outputting the processed video stream.

[0006] In a first aspect of this application, a real-time video rendering method is provided, the method comprising: Load the first video; For any first video frame in the first video: Determine the rendering perspective; Based on the first video frame and the rendering perspective, the second video frame is rendered. In response to view control events, adjust the rendering viewpoint to serve as the rendering viewpoint for the next first video frame.

[0007] According to embodiments of this application, the real-time video rendering method further includes: The first video frame is analyzed based on the tracking algorithm to obtain the view control event.

[0008] Preferably, the method further includes, in the initial state, analyzing the first video frame in the first video based on the tracking algorithm, and obtaining the rendering perspective in the initial state based on the initial tracking points output by the tracking algorithm.

[0009] Preferably, the viewpoint control event is obtained by analyzing the first video frame based on the tracking algorithm, including: The tracking points are obtained by analyzing the first video frame using a tracking algorithm. Calculate the offset based on the location of the tracking point; The instruction to control the rendering viewpoint by reducing the offset is used as the viewpoint control event.

[0010] Preferably, based on the first video frame and the rendering perspective, a second video frame is rendered, including: The first video frame is projected onto the sphere to obtain the spherical image; The second video frame is rendered from the spherical image based on the rendering perspective, preset field of view, and preset frame size.

[0011] Preferably, the offset is calculated based on the position of the tracking point, including: The center point of the second video frame is projected onto the first video frame to obtain the reference point; Calculate the offset between the reference point and the tracking point to obtain the offset amount.

[0012] In a second aspect of this application, a video processing method is provided, the method comprising: For each first video frame in the first video, the method described in the first aspect of this application is executed sequentially to obtain each second video frame corresponding to each first video frame; The processed second video is composed of all the second video frames.

[0013] In a third aspect of this application, a mapping system is provided, comprising: The loading subsystem is suitable for loading the first video. The rendering subsystem is adapted to determine the rendering perspective for any first video frame in the first video, render a second video frame based on the first video frame and the rendering perspective, and adjust the rendering perspective in response to the perspective control event as the rendering perspective for the next first video frame.

[0014] In a fourth aspect of this application, a video processing apparatus is provided, comprising: The loading module is suitable for loading the first video; The rendering module is adapted to determine the rendering perspective for any first video frame in the first video, render the second video frame based on the first video frame and the rendering perspective, and adjust the rendering perspective in response to the perspective control event as the rendering perspective for the next first video frame. The processing module is adapted to render each first video frame in the first video based on the rendering module to obtain each corresponding second video frame, and to combine all the second video frames into a processed second video.

[0015] In a fifth aspect of this application, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the methods of the first and second aspects of this application.

[0016] The real-time video rendering method of this application effectively solves the problems of real-time analysis and rendering of wide-angle videos and low user interactivity in existing technologies. This system can synchronously analyze and adjust the perspective of real-time captured wide-angle video footage and output rendered images. It can also output rendered images based on user-specified perspective changes, ultimately resulting in a complete and continuous processed video stream. This real-time video processing flow based on preset processing logic or user-specified logic not only significantly expands the applicable scenarios of wide-angle video rendering methods and improves scene adaptability and processing timeliness, but also enhances the flexibility of wide-angle video rendering output perspective and the utilization rate of video data resources according to users' personalized needs, demonstrating significant technical effectiveness. Attached Figure Description

[0017] The above and other features, advantages, and aspects of the embodiments of this application will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein: Figure 1 This is a flowchart of a real-time video rendering method according to an embodiment of this application; Figure 2 This is a schematic diagram of an interface for controlling events from a human input perspective, according to an embodiment of this application. Figure 3 This is a flowchart of a video processing method according to an embodiment of this application; Figure 4 This is a block diagram of a mapping system according to an embodiment of this application; Figure 5 This is a block diagram of a video processing apparatus according to an embodiment of this application; Figure 6 This is a schematic diagram of the structure of a computer device suitable for implementing the embodiments of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] In the field of computer vision analysis, the origin of the coordinate system is usually located at the upper left corner of the screen by default. In all embodiments of this application, unless otherwise stated, the upper left corner is used as the origin of the screen coordinate system. Those skilled in the art should know that such a coordinate system setting is not absolutely fixed. When the origin of the coordinate system is set at any position inside or outside the screen, the corresponding technical solutions that can be obtained by simple adjustments to this solution without creative effort are all within the protection scope of this application.

[0020] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0021] The real-time video rendering method provided in this application can run on various computing devices, including laptops, smartphones, tablets, or other smart devices with data acquisition capabilities, such as smart cameras and wearable devices.

[0022] According to one embodiment of this application, a real-time video rendering method is provided, the purpose of which is to crop videos captured by a wide field-of-view device into videos that simulate those captured by a general field-of-view device. In the subject matter of this application, the video can be represented as static video data or as a dynamic video stream. When the input is static video data, this method can process video frames continuously, frame by frame; when the input is a dynamic video stream, this method can process real-time acquired video frames continuously, frame by frame. Therefore, this method can achieve analysis and processing of static video data as well as real-time or near-real-time analysis and processing of continuous dynamic video streams. Using a wide field-of-view device to capture video is typically to capture the entire scene; the device can be a wide-angle lens or a combination of multiple lenses, or a 360° panoramic video capture device.

[0023] In the context of this application, rendering refers to the process of analyzing video footage and then regenerating one or more new video frames. The rendering in this application's embodiments can be implemented based on spatial geometric reconstruction through projection transformation, or based on cross-dimensional coordinate mapping of viewpoint parameters. This application does not limit the algorithm logic or hardware environment used for rendering. Rendering can effectively restore the distortion of images captured with a large field of view, outputting video footage that conforms to the perspective laws of the human eye.

[0024] Meanwhile, based on the video's presentation format, the real-time rendering described in this application can be a frame-by-frame synchronous or near-synchronous rendering process based on the acquisition time of the real-time acquired wide-angle video frames, or a frame-by-frame synchronous or near-synchronous rendering process based on the reading time of stored or uploaded static video data. Real-time rendering enables the system to output the processed video image in real time, significantly expanding the applicable scenarios of wide-angle video processing technology and improving processing efficiency.

[0025] Example 1:

[0026] Figure 1 A flowchart of a real-time video rendering method according to an embodiment of this application is shown, as follows: Figure 1 As shown: First, the first video is loaded. In this application, the first video refers to the video in the subject matter. Its definition and acquisition device have been explained in detail above and will not be repeated here. Loading refers to processing the first video to obtain all the first video frames. In one possible implementation, the system first needs to acquire the first video. This could be acquiring first video data transmitted from an external source to the system or already stored within the system, or acquiring first video footage captured in real-time or first video footage transmitted and displayed in real-time. Then, the acquired first video is read through a video decoding library, decoded frame by frame according to its frame rate, and each frame of image data is saved as an independent first video frame.

[0027] Secondly, for any first video frame in the first video, the following steps are performed: S1: Determine the rendering perspective; S2: Render a second video frame based on the first video frame and the rendering perspective; S3: In response to the view control event, adjust the rendering view as the rendering view of the next first video frame.

[0028] The steps S1-S3 described above are steps that sequentially process video frames in a loop. In S2, rendering is performed based on the rendering perspective obtained in S1; simultaneously, S3 processes the first video frame to obtain a new rendering perspective, which is used as the rendering perspective obtained in S1 when performing the same processing step on the next video frame. That is, this method can perform loop processing on the first video frame that needs to be processed in the first video according to the process, and while rendering the current frame based on the rendering perspective calculated when processing the previous frame, it also calculates the rendering perspective used to render the next frame.

[0029] The first video frame refers to a specific video frame in the first video. Any first video frame can be a video frame extracted at a specified or fixed interval in the initial state, such as extracting one frame every two frames; or it can be a first video frame obtained sequentially, such as the next frame adjacent to the previous first video frame or a video frame that needs to be processed after the previous first video frame. That is, this application can execute this method based on the difference in the selection of the first video frame.

[0030] The processing of any first video frame in the first video is based on the following considerations: First, in practical application scenarios, taking a football match as an example, a professional football match may involve extra time, penalty kicks, etc. Recording a complete match video results in a long video length, a huge and complex amount of data, and may include non-exciting segments such as halftime breaks and pre- and post-match ceremonies. These non-exciting segments have low value in actual match analysis and low viewer interest. Therefore, when analyzing and processing video footage, these segments are secondary or do not require computational processing. The selection of the first video frame can skip these segments, such as specifying video frames other than these segments in the initial state, or skipping these segments during the loop phase to select the next frame to be processed. This improves the utilization efficiency of computing resources and enhances the flexibility and efficiency of the system's data analysis and processing. Second, in this method, as mentioned above, any first video frame includes the case where it is the next first video frame after the previous first video frame. By describing the connection between consecutive first video frames, this method can be executed cyclically to achieve the technical objectives and effects of this application.

[0031] S1: Determine the rendering perspective.

[0032] As mentioned earlier, rendering refers to the process of analyzing video footage and then regenerating one or more new video frames. The rendering perspective refers to the orientation and field of view of the first video frame in a 3D spatial coordinate system. The rendering perspective defines the projection reference for mapping three-dimensional spherical pixels to two-dimensional planar pixels. Through this determined mapping relationship, the corresponding pixels in the first video frame can be restored to a second video frame that conforms to the perspective laws of the human eye, making the output image viewable.

[0033] Determining the rendering perspective can be the rendering perspective of the first video frame extracted at a specified or fixed interval in the initial state, such as extracting one frame every two frames; or it can be the rendering perspective of the first video frames obtained in sequence, such as the rendering perspective of the next frame adjacent to the previous first video frame or the rendering perspective of the video frames that need to be processed after the previous first video frame.

[0034] Specifically, when the first video frame processed in this step is the first frame of the first video, the step of determining the rendering perspective is the determination of the rendering perspective in the initial state.

[0035] In one possible implementation, the initial rendering viewpoint is specified by the user. The user determines the initial rendering viewpoint through a viewpoint specification operation. Viewpoint specification operations include, but are not limited to, user input of viewpoint specification code, user performance of viewpoint specification actions on the controller, etc. All methods that allow the user to specify the rendering viewpoint are within the scope of this application. Considering that in the initial state, users may prefer to start watching the video from a specific perspective—for example, in a football match video, a user may prefer to watch a specific team or player; in a traffic monitoring video, a user may prefer to watch a specific vehicle or pedestrian—allowing the user to specify the initial rendering viewpoint aims to improve the system's user interactivity, meet users' personalized viewing needs, and improve the system's output flexibility and targeting.

[0036] In another possible implementation, the initial rendering perspective is the initial zero-degree perspective. The initial zero-degree perspective refers to the gimbal yaw angle when the system starts shooting or the perspective of the first frame of video data that needs to be processed. Choosing the zero-degree perspective as the initial rendering perspective is conventional, can save computing power, and improve the stability of video processing.

[0037] In another possible implementation, the initial rendering viewpoint is determined centered on the point of highest target density in the initial state image. A target refers to an object of interest in the image that requires spatial relationship analysis. Targets can be objects of the same type or similar objects with high correlation in the image, such as players and balls. Targets can be obtained by analyzing the initial state image using object detection models, such as YOLO (You Only Look Once) and SSD (Single Shot MultiBox Detector). The output detection results include the target detection boxes in the image, typically represented by four parameters (x, y, width, height), where x and y represent the horizontal and vertical coordinates of the target detection box's center point, respectively; width represents the width of the target detection box; and height represents the height of the target detection box. Based on the coordinates (x, y) of the center point of each target detection box, density clustering algorithms can be used to cluster the targets, with the center of the main cluster being taken as the point of highest target density. Subsequently, the highest point of the target distribution density was inversely projected into the corresponding coordinates in a spherical coordinate system using an equidistant cylindrical projection model; based on the inverse projection coordinates and the coordinates of the sphere center, the target azimuth (ϕ) was obtained. target ,θ target ); Get the current azimuth angle (ϕ) of the initial optical axis in 3D space when not in use. current ,θ currentBased on the yaw angle difference Δϕ and pitch angle difference Δθ between the target azimuth and the current azimuth, the current azimuth is controlled to rotate towards the target azimuth. The azimuth angle after the two coincide is the initial rendering viewpoint determined with the highest point of target distribution density as the center. The purpose of this method is to focus the viewpoint of the output screen on the core area with higher target concentration and activity from the initial state, thereby improving the scene semantic relevance of the output screen and enhancing the user viewing experience.

[0038] In a preferred implementation, in the initial state, the initial rendering perspective is obtained by analyzing the first video frame in the first video based on the tracking algorithm, and then using the initial tracking point output by the tracking algorithm. It can be understood that the basic logic of the tracking algorithm is to determine whether the shooting perspective should be adjusted to continuously track and capture key targets in real time by observing the movement of targets in consecutive frames captured by the shooting device. For each new input target detection result, the tracking algorithm may calculate a new tracking strategy. In this implementation, the tracking algorithm needs to analyze the first video frame in the first video to obtain the coordinates of the key target to be tracked, i.e., the initial tracking point. The specific choice of tracking algorithm is not specifically limited in this application. Generally, mature target tracking algorithms such as MHT, MOT, and SORT can be used to implement the technical solution of this application. Specifically, the initial tracking point can be the coordinates of the center point of the target detection box of a specific target, such as the ball in a football match scene or the fastest moving human target in the frame. The initial rendering perspective is obtained based on the initial tracking point. In one possible implementation, the initial rendering perspective is determined with the initial tracking point as the center. Specifically, the initial tracking point is inversely projected into corresponding coordinates in a spherical coordinate system using an equidistant cylindrical projection model; based on the inverse projection coordinates and the coordinates of the sphere's center, the target azimuth angle (ϕ) is obtained. target’ ,θ target’ ); Get the current azimuth angle (ϕ) of the initial optical axis in 3D space when not in use. current’ ,θ current’ Based on the yaw angle difference Δϕ and pitch angle difference Δθ between the target azimuth and the current azimuth, the current azimuth is controlled to rotate towards the target azimuth. The azimuth after the two coincide is the rendering viewpoint in the initial state, centered on the initial tracking point. This method enables the viewpoint of the output screen to be focused on the tracking point analyzed by the tracking algorithm in the initial state, improving the scene semantic relevance of the output screen and enhancing the user's viewing experience.

[0039] S2: Based on the first video frame and the rendering perspective, a second video frame is rendered.

[0040] The second video frame refers to the frame obtained by rendering the first video frame, which conforms to the perspective laws of the human eye. Because the first video frame is captured by a wide field-of-view device, the image presented has spatial distortion and does not conform to the perspective laws of the human eye. Therefore, it cannot be directly used as a display image. It is necessary to render the first video frame to form a second video frame that is easy for users to view directly and conforms to the perspective laws of the human eye.

[0041] In one possible implementation, the rendering of the second video frame can be performed by a pre-trained image generation model. This model takes the first video frame and the rendering perspective as joint input, and uses a neural network to reconstruct the first video frame based on latent space feature transformation. Specifically, it uses the neural network to compress the original image to low-dimensional latent space features, and combines this with the rendering perspective to perform feature alignment and perspective transformation within this latent space. Finally, a decoding network reconstructs and outputs the second video frame. This rendering method can output video frames that conform to the perspective laws of the human eye, and can use deep learning algorithms to predict and complete any missing details in the first video frame, thereby optimizing the clarity of the output video frame and the user's viewing experience.

[0042] S3: In response to the view control event, adjust the rendering view as the rendering view of the next first video frame.

[0043] View control events refer to events that drive changes in the rendering view. These can be operations that control changes in the rendering view, such as user input of view specification code or user-defined view actions on the controller. They can also be synchronous adjustments to the rendering view triggered by the offset of the tracking point position captured by the tracking algorithm.

[0044] The rendering perspective is adjusted to serve as the rendering perspective for the next first video frame. This next first video frame can be the frame immediately following the first video frame, or it can be the next video frame that needs to be processed after the first video frame. Therefore, the rendering perspective of the next first video frame can be either the frame immediately following the first video frame or the next video frame that needs to be processed after the first video frame. This operation updates the rendering perspective, allowing the next first video frame to render a second video frame from a different perspective. This process iteratively processes the first video frames that need to be processed within the first video, enabling the method to execute cyclically according to the flow. In one possible implementation, the user adjusts the rendering perspective by specifying the perspective. Figure 2 The diagram illustrates an interface for controlling events from a human input perspective, according to an embodiment of this application. Figure 2 As shown. Specifically, users can control the rendering perspective by inputting perspective specification codes or performing perspective specification actions on the controller, which will serve as the rendering perspective for the next first video frame.

[0045] Example 2:

[0046] In a preferred implementation, the viewpoint control event in Example 1 can be automatically obtained based on the analysis of the first video frame using a tracking algorithm. As mentioned above, the basic logic of the tracking algorithm is to determine whether the shooting viewpoint should be adjusted to continuously track and capture key targets in real time by analyzing the anomalies of targets in continuous frames captured by the shooting device. For each newly input target detection result in each frame, the tracking algorithm may calculate a new tracking strategy. In this implementation, the tracking algorithm obtains the tracking point by analyzing the first video frame and predicts the rendering viewpoint change path based on the displacement law of the tracking point between multiple frames, thereby obtaining the viewpoint control event. The specific selection of the tracking algorithm is not specifically limited in this application. Generally, mature target tracking algorithms such as MHT, MOT, and SORT can be used to implement the technical solution of this application. First, the tracking algorithm needs to determine the tracking point, that is, under the analysis results of the current first video frame, the tracking algorithm believes that the video frame should be adjusted to be centered on which tracking point. The tracking point can be the coordinates of the highest point of target distribution density in the first video frame, or the coordinates of the center point of the target detection box of a specific target, such as the ball target in a football match scene or the fastest human target in the picture.

[0047] Secondly, the tracking algorithm needs to predict the path of changes in the rendering viewpoint. Preferably, after determining the tracking point, the SORT algorithm is used to track the tracking point. SORT (Simple Online and Realtime Tracking) is an efficient multi-target tracking algorithm based on motion modeling and data association. Its core idea is to achieve continuous tracking of the target identity across frames by fusing the target motion prediction in the time dimension and the detection box position information in the spatial dimension. First, Kalman filtering is used to predict the bounding box position of the tracking point in the next first video frame based on its historical trajectory. Then, the Hungarian algorithm is introduced to establish the association between the current first video frame containing the tracking point and the previous first video frame containing the tracking point. By minimizing the association cost, the tracking point in the current first video frame is matched with the tracking point in the previous first video frame, ensuring that the bounding box of each tracking point is associated with at most one predicted bounding box, thereby obtaining the next predicted position of the tracking point. Specifically, if the position of the tracking point in the first video frame is (x1, y1), and the predicted position of the tracking point in the next first video frame obtained by the multi-target tracking algorithm is (x2, y2), then the yaw angle Δϕ that needs to be adjusted in the rendering viewpoint is:

[0048] Where D is the target horizontal parameter. The yaw angle Δθ that needs to be adjusted for the rendering viewpoint is:

[0049] In addition, the rotation speed of the rendering viewpoint can be controlled by a PID controller. A smooth rotation speed command can be generated based on the pitch or yaw angle difference to prevent the rendering viewpoint from rotating too fast or too slow at any given moment. The formula for calculating the smooth rotation speed u(t) is as follows:

[0050] Where e(t) is the pitch angle difference or yaw angle difference, K p To set the hyperparameter to a value of 0.8, K i The hyperparameter is set to 0.1. This method analyzes the tracking points and predicts the path of the rendering viewpoint based on the displacement patterns of the tracking points. It uses predicted coordinates to reduce system processing latency, and uses a PID controller to improve the smoothness and stability of the output image, thereby enhancing the robustness of the system.

[0051] In another possible implementation, the tracking algorithm output includes the position coordinates of the tracking point, its velocity, and direction of motion in the current first video frame. The position coordinates of the tracking point can be the coordinates of the highest point in the target distribution density of the first video frame, or the coordinates of the center point of the target detection bounding box for a specific target. The system directly maps the velocity and direction of the tracking point to the rotational angular velocity command of the rendering viewpoint, as a viewpoint control event. This method achieves synchronous movement between the rendering viewpoint and the tracked target by keeping the rotational velocity and direction of the rendering viewpoint consistent with the movement velocity and direction of the tracking point. This method proactively eliminates positional deviations by adjusting the viewpoint synchronously when the tracking point shifts, rather than passively eliminating them after they occur. This reduces system response latency and makes the transition between the second and third video frames smoother and more stable.

[0052] Preferably, in another possible implementation, the viewpoint control event is obtained by analyzing the first video frame based on a tracking algorithm, including: Based on the tracking algorithm, the tracking points are obtained by analyzing the first video frame. Calculate the offset based on the position of the tracking point; The instruction to control the rendering viewpoint by reducing the offset is used as the viewpoint control event.

[0053] The specific implementation method for obtaining tracking points based on the tracking algorithm analysis of the first video frame has been described in detail above and will not be repeated here. The offset is calculated based on the position of the tracking point. The offset in this method includes two aspects: the distance between two specific position coordinates and the offset direction. Specifically, in one possible implementation, the offset is calculated based on the position coordinates of the tracking point in the previous first video frame and the position coordinates of the tracking point in the current first video frame. Preferably, in another possible implementation, calculating the offset based on the position of the tracking point includes the following two steps: The center point of the second video frame is projected onto the first video frame to obtain the reference point; Calculate the offset between the reference point and the tracking point to obtain the offset amount.

[0054] The center point of the second video frame refers to the geometric center of the second video frame, corresponding to the optical axis direction of the rendering viewpoint of the first video frame. Specifically, projection involves mapping the center point of the second video frame back to its original corresponding position in the first video frame through reverse perspective, obtaining the reference point. The offset between the reference point and the tracking point is calculated to obtain the offset amount. In one possible implementation, the offset amount is calculated based on the position coordinates of the reference point and the position coordinates of the tracking point. Based on the first implementation, the reference point is obtained based on the rendering viewpoint of the first video frame, which in turn is obtained by adjusting the rendering viewpoint of the previous first video frame. Specifically, when adjusting the rendering viewpoint of the previous first video frame, there may be a short frame interval or a large offset of the tracking point. The rendering viewpoint has already entered the next frame while turning towards the tracking point. At this time, the rendering viewpoint of the next frame is determined based on the rendering viewpoint during the adjustment process, rather than based on the tracking point of the previous frame. The advantage of this approach is that it can quickly obtain the precise offset between the rendering viewpoint and the tracking point, even when the inter-frame interval is short or the tracking point is significantly offset. This reduces system computing power consumption and running latency, improves the timeliness of outputting subsequent video frames, and enhances the user's viewing experience.

[0055] The instruction to control the rendering viewpoint by reducing the offset is considered a viewpoint control event. In one possible implementation, the Euclidean distance between the coordinates of the reference point and the tracking point is calculated, and the straight line from the reference point to the tracking point is used as the offset direction. The instruction to control the rendering viewpoint to move in the offset direction, reducing the Euclidean distance, until the optical axis of the rendering viewpoint points to the tracking point, is the viewpoint control event.

[0056] Based on Embodiment 1, in a preferred implementation, a second video frame is rendered based on the first video frame and the rendering viewpoint, including: The first video frame is projected onto the sphere to obtain the spherical image; The second video frame is rendered from the spherical image based on the rendering perspective, preset field of view, and preset frame size.

[0057] Projecting the first video frame onto a sphere can be achieved by inverse projection of an equidistant cylindrical projection, mapping the first video frame back to a three-dimensional spherical coordinate system. Specifically, each pixel in the first video frame is traversed to obtain pixel coordinates (u, v), and each pixel coordinate is normalized to obtain normalized planar coordinates (U, v). norm V norm ):

[0058] Where W and H are the horizontal and vertical resolutions of the first video frame, respectively, and +0.5 is to ensure that the sampling point of the pixel coordinates is located at the geometric center of the pixel grid, avoiding sampling gaps at the seams of the sphere.

[0059] The normalized planar coordinates are mapped to longitude λ and latitude ϕ on a sphere. For ease of subsequent trigonometric function calculations, radians are used uniformly here.

[0060] Based on the obtained spherical latitude and longitude (λ, ϕ), according to the right-hand coordinate system standard, calculate the three-dimensional spatial coordinates (x, y, z) of each pixel on a unit sphere with a radius of 1:

[0061] The calculated 3D spatial coordinates (x, y, z) are set as the vertex position data of the spherical mesh, and the normalized coordinates (U... norm V norm The UV texture coordinates of the corresponding vertices are encapsulated, and the shader samples the first video frame based on the UV coordinates. Through the built-in bilinear interpolation and accelerated rendering technology of the graphics hardware, the system can automatically handle the smooth transition of colors between non-integer coordinates, and finally generate a continuous and seamless spherical image in three-dimensional space.

[0062] Based on the rendering perspective, preset field of view, and preset frame size, a second video frame is rendered from the spherical image. This process simulates the shooting behavior of a virtual camera at the center of the sphere, projecting the spherical image information into a second video frame that conforms to the perspective laws of the human eye. Specifically, the optical axis of the rendering perspective determines the geometric center of the second video frame, the preset field of view determines the zoom level of the second video frame, and the preset frame size determines the resolution of the second video frame. In detail, after determining the position of the virtual imaging plane in 3D space based on the rendering perspective, preset field of view, and preset frame size, the system traverses every pixel on the imaging plane, emitting rays outward from the center of the sphere and calculating the coordinates of their intersection points with the sphere. The intersection coordinates are then traced back to the corresponding UV coordinates in the first video frame for color sampling, and the extracted color values ​​are filled into the corresponding positions on the imaging plane, ultimately obtaining the second video frame.

[0063] Since the first video frame is a distorted wide-angle shot, it's impossible to directly capture a second video frame suitable for human viewing. This rendering method uses a spherical image as an intermediary to output video frames that conform to the perspective laws of the human eye, ensuring that the image doesn't break or shift when the rendering viewpoint passes through seams. Furthermore, by changing the preset field of view, a smooth transition from a wide-angle lens to a telephoto lens can be simulated, enhancing the system's compositional flexibility and interactive freedom.

[0064] Figure 3 A flowchart illustrating a video processing method according to an embodiment of this application is shown, such as... Figure 3 As shown: For each first video frame in the first video, the method as described in any one of claims 1-6 is executed sequentially to obtain each second video frame corresponding to each first video frame; The processed second video is composed of all the second video frames.

[0065] According to the methods in Embodiments 1 and 2, after obtaining each corresponding second video frame based on each first video frame to be processed in the first video, the processed second video is composed of all the second video frames. In one possible implementation, all the second video frames are arranged and spliced ​​in ascending order of their timestamps in the first video. Specifically, if adjacent second video frames are consecutive after ascending order or the timestamp difference is less than or equal to a first threshold, the adjacent frames are directly spliced; if the timestamp difference between adjacent second video frames is greater than the first threshold, a cross-fade-in / fade-out transition effect is inserted at the boundary of the adjacent frames, and during the transition, the previous second video frame is subjected to slow blur processing, and the subsequent second video frame is subjected to synchronous transparency gradient processing. The first threshold is dynamically determined according to the frame rate of the first video. For example, if the first video is a standard streaming video at 30FPS, the first threshold can be set to 0.2s according to the persistence of vision of the human eye; if the first video is a high dynamic range competitive sports video at 60FPS, the first threshold can be set to 0.1s to adapt to the high-speed changes in the picture. Finally, the output is a cropped second video with matching transition effects. It's understandable that if each frame of the first video is extracted at a fixed interval, the frame rate of the second video will be proportionally reduced; specifically, it's inversely proportional to the extraction interval. For example, if the first video is 10 seconds long, has a frame rate of 20 FPS, and a total of 200 frames, and uses a fixed extraction interval of 1 frame every 2 frames, the resulting second video will have a total of 100 frames and the same duration of 10 seconds. In this case, the frame rate of the second video will become 10 FPS, inversely proportional to the extraction interval.

[0066] This method allows for flexible processing of second video frames based on the different types of the first video. When the inter-frame interval is small, the human eye's visual temporary storage compensation mechanism is used to avoid transition processing, ensuring smooth viewing. When the inter-frame interval is large, transition effects are incorporated to reduce the impact of large changes in perspective or discontinuities in the preceding and following scenes on the user's viewing experience. Simultaneously, the final output second video retains all second video frame data generated based on the tracking algorithm or user-specified rendering, facilitating repeated viewing and analysis later.

[0067] Figure 4 This is a block diagram of a mapping system 300 according to an embodiment of this application. Figure 4 As shown, the mapping system 300 includes a loading subsystem 301 and a rendering subsystem 302.

[0068] The loading subsystem 301 is adapted to load a first video. In one possible embodiment, the subsystem can read and process video data provided internally by the system; in another embodiment, the subsystem can receive and process video data provided by other devices or interfaces. The loading subsystem 301 is adapted to load the first video and send it to the rendering subsystem 302. Optionally, the loading subsystem 301 may not have an active information sending function, and instead, the rendering subsystem 302 may actively obtain the first video from the loading subsystem 301.

[0069] The rendering subsystem 302 is adapted to determine a rendering perspective for any first video frame in the first video, render a second video frame based on the first video frame and the rendering perspective, and adjust the rendering perspective in response to a perspective control event as the rendering perspective for the next first video frame. The rendering subsystem 302 can be implemented as an executable program or code in a smart terminal or server.

[0070] The specific methods for the execution of each subsystem of this system have been described in detail in the aforementioned corresponding embodiments, and will not be repeated here.

[0071] Figure 5 This is a block diagram of a video processing apparatus 400 according to an embodiment of this application. Figure 5 As shown, the video processing device 400 includes a loading module 401, a rendering module 402, and a processing module 403.

[0072] The loading module 401 is adapted to load a first video. In one possible embodiment, the module can read and process video data provided internally by the system; in another embodiment, the module can receive and process video data provided by other devices or interfaces. The loading module 401 is adapted to load the first video and send it to the rendering module 402. Optionally, the loading module 401 may not have an active information sending function, and instead, the rendering module 402 may actively obtain the first video from the loading module 401.

[0073] The rendering module 402 is adapted to determine a rendering perspective for any first video frame in the first video, render a second video frame based on the first video frame and the rendering perspective, and adjust the rendering perspective in response to a perspective control event as the rendering perspective for the next first video frame. Optionally, the rendering module 402 may not have an active information sending function; instead, the processing module 403 may actively obtain the second video frame from the rendering module 402. The rendering module 402 can be implemented as an executable program or code in a smart terminal or server.

[0074] The processing module 403 is adapted to render each corresponding second video frame based on each first video frame in the first video by the rendering module, and to combine all the second video frames into a processed second video. The processing module 403 can be implemented as an executable program or code in a smart terminal or server.

[0075] The specific methods for each module of this device have been described in detail in the aforementioned corresponding embodiments, and will not be repeated here.

[0076] Figure 6 A schematic diagram of a computer device suitable for implementing embodiments of this application is shown. The computer device can be implemented as a terminal device or a server.

[0077] like Figure 6 As shown, the terminal device or server includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 502 or a program loaded from storage section 508 into random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the terminal device or server. The CPU 501, ROM 502, and RAM 503 are interconnected via bus 504. An input / output (I / O) interface 505 is also connected to bus 504.

[0078] The following components are connected to I / O interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to I / O interface 505 as needed. A removable medium 511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 510 as needed so that computer programs read from it can be installed into storage section 508 as needed.

[0079] Specifically, according to embodiments of this application, the above method flow steps can be implemented as a computer software program. For example, embodiments of this application include a computer program product comprising a computer program carried on a machine-readable medium, the computer program containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by central processing unit (CPU) 501, it performs the functions defined in the system of this application.

[0080] It should be noted that the computer-readable medium shown in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, or any suitable combination thereof.

[0081] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0082] The units or modules described in the embodiments of this application can be implemented in software or hardware. The described units or modules can also be located in a processor. The names of these units or modules do not, in certain circumstances, constitute a limitation on the unit or module itself.

[0083] In another aspect, this application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable storage medium stores one or more programs that are used by one or more processors to execute the methods described in this application.

[0084] In another aspect, embodiments of this application also provide a computer program product that, when executed by a processor, implements the methods of any of the above embodiments.

[0085] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the foregoing application concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions claimed in this application.

Claims

1. A real-time video rendering method, characterized in that, The method includes: Load the first video; For any first video frame in the first video: Determine the rendering perspective; Based on the first video frame and the rendering perspective, a second video frame is rendered; In response to a view control event, the rendering view is adjusted as the rendering view for the next first video frame.

2. The method as described in claim 1, characterized in that, Also includes: The view control event is obtained by analyzing the first video frame based on the tracking algorithm.

3. The method as described in claim 1 or 2, characterized in that, Also includes: In the initial state, the first video frame in the first video is analyzed based on the tracking algorithm, and the rendering viewpoint in the initial state is obtained based on the initial tracking points output by the tracking algorithm.

4. The method as described in claim 2, characterized in that, The step of analyzing the first video frame based on a tracking algorithm to obtain the viewpoint control event includes: Based on the tracking algorithm, the tracking points are obtained by analyzing the first video frame. Calculate the offset based on the position of the tracking point; The instruction to control the rendering viewpoint by reducing the offset is used as the viewpoint control event.

5. The method of claim 1, wherein, Based on the first video frame and the rendering perspective, a second video frame is rendered, including: The first video frame is projected onto the sphere to obtain a spherical image; Based on the rendering perspective, preset field of view, and preset frame size, a second video frame is rendered from the spherical image.

6. The method as described in claim 5, characterized in that, The calculation of the offset based on the position of the tracking point includes: The center point of the second video frame is projected onto the first video frame to obtain the reference point; The offset between the reference point and the tracking point is calculated to obtain the offset amount.

7. A video processing method, characterized in that, The method includes: For each first video frame in the first video, the method as described in any one of claims 1-6 is executed sequentially to obtain each second video frame corresponding to each first video frame; The processed second video is composed of all the second video frames.

8. A mapping system, characterized in that, include: The loading subsystem is suitable for loading the first video. The rendering subsystem is adapted to determine a rendering perspective for any first video frame in the first video, render a second video frame based on the first video frame and the rendering perspective, and adjust the rendering perspective as the rendering perspective for the next first video frame in response to a perspective control event.

9. A video processing apparatus, characterized in that, include: The loading module is suitable for loading the first video; The rendering module is adapted to determine the rendering perspective for any first video frame in the first video, render a second video frame based on the first video frame and the rendering perspective, and adjust the rendering perspective in response to a perspective control event as the rendering perspective for the next first video frame. The processing module is adapted to render each second video frame corresponding to each first video frame in the first video based on the rendering module, and to combine all the second video frames into a processed second video.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-7.

Images