Data processing method and system

Abstract

The embodiment of the present specification provides a data processing method and system, the method comprises: acquiring a plurality of videos of a target object under a plurality of perspectives; acquiring key video frames of each video from the plurality of videos under the plurality of perspectives; constructing a three-dimensional model of the target object at the key video frames according to the object parameters of the target object in the key video frames of each video; sending the three-dimensional model of the target object at the key video frames and the position information of the key video frames and other video frames to a client, so that the client obtains the three-dimensional model of the target object at other video frames based on the three-dimensional model and the position information, and obtains a target video according to the three-dimensional model of the target object at the key video frames and the three-dimensional model at other video frames; constructing and sending the three-dimensional model of the target object at the key video frames, which reduces data redundancy, improves storage efficiency, reduces the amount of data to be transmitted, and reduces the occupation of network bandwidth.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of computer technology, and in particular to a data processing method. One or more embodiments of this specification also relate to another data processing method, a data processing system, a computing device, an electronic device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] In today's technology-driven world, high-quality broadcasts of major events such as sporting events and concerts are crucial for the audience experience. This is not only about providing clear visuals, but also about creating an immersive experience for viewers. To achieve this, scientists and engineers are constantly working to improve video transmission and display technologies, especially in the capture and reconstruction of highly dynamic four-dimensional (4D) scenes.

[0003] 4D scene capture extends beyond three-dimensional space to include changes over time, recording dynamic changes from multiple angles and moments. This technology is crucial for live event broadcasts because it more accurately recreates the experience, providing viewers with a more realistic and interactive experience. However, capturing this data generates massive amounts of data, posing significant challenges to storage and transmission. Traditional methods have limitations when processing long sequence times, primarily due to the need for substantial storage and extremely high bandwidth, leading to a sharp increase in costs. Summary of the Invention

[0004] In view of this, embodiments of this specification provide a data processing method. One or more embodiments of this specification also relate to another data processing method, a data processing system, a computing device, an electronic device, a computer-readable storage medium, and a computer program product, to address the technical shortcomings of existing technologies that require large amounts of storage space and extremely high transmission bandwidth when processing scene sequences.

[0005] According to a first aspect of the embodiments of this specification, a data processing method is provided, comprising:

[0006] The target object is obtained from multiple videos from multiple perspectives. From the multiple videos from multiple perspectives, key video frames of each video are obtained. The multiple videos are time-synchronized, and the key video frames of each video correspond to the same time.

[0007] Based on the object parameters of the target object in the key video frames of each video, construct a three-dimensional model of the target object at the key video frame;

[0008] The 3D model of the target object at the key video frame and the position information of the key video frame and other video frames are sent to the client. The other video frames are the video frames in each video except for the key video frame. The 3D model and the position information are used to enable the client to obtain the 3D model of the target object at the other video frames, and to obtain the target video based on the 3D model of the target object at the key video frame and the 3D model of the target object at the other video frames.

[0009] According to a second aspect of the embodiments of this specification, another data processing method is provided, applied to a client, including:

[0010] Receive a 3D model of the target object at a key video frame, and positional information between the key video frame and other video frames, wherein the key video frame is a key video frame of the target object in multiple videos from multiple perspectives, and the other video frames are video frames in the multiple videos other than the key video frame;

[0011] The 3D model at the key video frame and the location information are input into the target parameter prediction network to obtain the 3D model of the target object at other video frames.

[0012] The target video is obtained based on the 3D model of the target object at the key video frame and the 3D model at other video frames.

[0013] According to a third aspect of the embodiments of this specification, a data processing system is provided, including a server and a client;

[0014] The server is configured to acquire multiple videos of a target object from multiple perspectives, and from these videos, extract key video frames for each video, wherein the multiple videos are time-synchronized and the key video frames correspond to the same time; construct a 3D model of the target object at the key video frame based on the object parameters of the target object in the key video frame; and send the 3D model of the target object at the key video frame, along with the positional information between the key video frame and other video frames, to the client, wherein the other video frames are all video frames in each video except for the key video frame.

[0015] The client is configured to receive a 3D model of the target object at a key video frame, as well as the location information of the key video frame and other video frames; input the 3D model at the key video frame and the location information into a target parameter prediction network to obtain a 3D model of the target object at other video frames; and obtain a target video based on the 3D model of the target object at the key video frame and the 3D model of the target object at other video frames.

[0016] According to a fourth aspect of the embodiments of this specification, a computing device is provided, comprising:

[0017] Memory and processor;

[0018] The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which implement the steps of the above data processing method when executed by the processor.

[0019] According to a fifth aspect of the embodiments of this specification, an electronic device is provided, comprising:

[0020] A memory and a processor, the memory and the processor being connected via a bus;

[0021] The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the above-described data processing method.

[0022] According to a sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program / instructions that, when executed by a processor, implement the steps of the data processing method described above.

[0023] According to a seventh aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the data processing method described above.

[0024] This specification provides a data processing method in one embodiment that acquires multiple videos of a target object from multiple perspectives, captures the movement or scene changes of the target object from different angles, and ensures that the multiple videos are time-synchronized, so the key video frames of each video correspond to the same time. Based on the object parameters of the target object in the key video frames of each video, a three-dimensional model of the target object at the key video frame is constructed. By using the object parameters in the key video frames from multiple perspectives, the three-dimensional model of the target object at the key video frame can be accurately constructed. By constructing a three-dimensional model at each key video frame, the dynamic changes of the target object in the time dimension can be captured, and data redundancy is greatly reduced, improving storage efficiency. When the three-dimensional model of the target object at the key video frame and the position information between the key video frame and other video frames are sent to the client, the amount of data to be transmitted is reduced, and the network bandwidth usage is reduced. Attached Figure Description

[0025] Figure 1 This is a schematic diagram illustrating a data processing method provided in one embodiment of this specification.

[0026] Figure 2 This is a flowchart illustrating a data processing method provided in one embodiment of this specification;

[0027] Figure 3 This is a flowchart illustrating a data processing method applied to a client, as provided in one embodiment of this specification.

[0028] Figure 4a This is a flowchart illustrating the data preprocessing process in a data processing method provided in one embodiment of this specification.

[0029] Figure 4b This is a flowchart of the dynamic reconstruction process in a data processing method provided in one embodiment of this specification;

[0030] Figure 4c This is a flowchart of the dynamic rendering process in a data processing method provided in one embodiment of this specification;

[0031] Figure 5 This is a schematic diagram of the structure of a data processing system provided in one embodiment of this specification;

[0032] Figure 6 This is a structural block diagram of a computing device provided in one embodiment of this specification;

[0033] Figure 7 This is a structural block diagram of an electronic device provided in one embodiment of this specification. Detailed Implementation

[0034] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0035] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0036] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0037] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0038] In one or more embodiments of this specification, a large model refers to a deep learning model with a large number of model parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even tens of trillions of model parameters. A large model can also be called a foundation model. It is pre-trained using large-scale unlabeled corpora to produce a pre-trained model with hundreds of millions of parameters. Such models can adapt to a wide range of downstream tasks and have good generalization ability. Examples include Large Language Models (LLMs) and multi-modal pre-training models.

[0039] In practical applications, large models only require a small number of samples to fine-tune the pre-trained model before they can be applied to different tasks. Large models can be widely used in fields such as Natural Language Processing (NLP) and Computer Vision. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and Image Generation, as well as natural language processing tasks such as text-based sentiment classification, text summarization, and machine translation. The main application scenarios for large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0040] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0041] 4D scene reconstruction refers to the data acquisition and modeling of a dynamic scene in three-dimensional space that changes over time, so as to visualize or analyze it in a computer environment.

[0042] 3D Gaussian Splatting: A technique for representing 3D point cloud data that expands 3D points into a Gaussian distribution in space, enabling modeling to better capture scene details.

[0043] Transformer network: A neural network based on attention mechanism for processing sequential information.

[0044] Keyframes: Important frames in animation or video that allow you to deduce the content of the entire sequence.

[0045] Position embedding maps positional information in input data to a feature space, and is typically used to preserve the order of elements in sequential data.

[0046] Existing 4D scene reconstruction technologies, especially in long sequence processing and highly dynamic scenes, have the following drawbacks: 4D technology generates a lot of redundant data and has low storage efficiency; it cannot be dynamically adjusted in real time: fixed segmentation strategies cannot adapt to changes in dynamic scenes of different intensities, resulting in a decrease in reconstruction accuracy or efficiency; and it has computational complexity: complex models lead to excessive computational overhead, affecting real-time performance.

[0047] To address these challenges, researchers are actively developing more advanced compression and reconstruction techniques. These techniques must not only reduce data volume to alleviate storage and transmission burdens, but also maintain high fidelity during decompression and reconstruction. This requires algorithmic innovation; simultaneously, developing effective real-time reconstruction techniques is crucial to ensure viewers can enjoy high-quality video without delay during transmission.

[0048] The introduction of cloud computing technology has offered new hope for solving storage and processing problems. By distributing complex data processing to the cloud, massive amounts of data can be managed and retrieved more effectively, further reducing reliance on local hardware. This integrated technology not only improves the efficiency of scene transmission and reconstruction but also opens up new possibilities for personalized display and immersive experiences.

[0049] To address the aforementioned technical problems, this specification provides a data processing method, and also relates to another data processing method, a data processing system, a computing device, an electronic device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0050] See Figure 1 , Figure 1 A schematic diagram of a data processing method according to an embodiment of this specification is shown.

[0051] Specifically, the data processing method is applied to a data processing system, which includes a client 102 and a server 104. In practice, multiple video feeds of the target object are obtained synchronously using multiple image acquisition devices from different perspectives, wherein the multiple video feeds are time-synchronized.

[0052] Server 104 acquires multiple videos of the target object from multiple perspectives, and obtains key video frames from each video from the multiple videos from multiple perspectives. The key video frames of each video correspond to the same time. Based on the object parameters of the target object in the key video frames of each video, a three-dimensional model of the target object at the key video frame is constructed. The three-dimensional model of the target object at the key video frame and the position information between the key video frame and other video frames are sent to client 102. The other video frames are the video frames in each video other than the key video frame.

[0053] Client 102 receives a 3D model of the target object at a key video frame and position information between the key video frame and other video frames; inputs the 3D model at the key video frame and the position information into a target parameter prediction network to obtain a 3D model of the target object at the other video frames; and obtains the target video based on the 3D model of the target object at the key video frame and the 3D model at the other video frames.

[0054] Client 102 may include a browser, an app (application), or a web application such as an H5 (Hypertext Markup Language 5) application, a lightweight application (also known as a mini-program), or a cloud application. The client device can be developed based on the software development kit (SDK) provided by the server, such as a real-time communication (RTC) SDK. The client device can be deployed in an electronic device and depends on the device's operation or certain apps running on the device. The electronic device may have a display screen and support information browsing, such as a personal mobile terminal like a mobile phone, tablet, or personal computer. Various other types of applications can also be configured in the electronic device, such as human-computer interaction applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, and social media platform software.

[0055] Server 104 can be understood as a server providing various services, including physical servers and cloud servers. Examples include servers providing communication services to multiple clients, servers supporting backend training of models used on clients, and servers processing data sent by clients. It's important to note that Server 104 can be implemented as a distributed server cluster composed of multiple servers, or as a single server. Server 104 can also be a server in a distributed system, or a server integrated with blockchain. Server 104 can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0056] This specification provides a data processing method in one embodiment. A server acquires multiple videos of a target object from multiple perspectives, capturing the movement or scene changes of the target object from different angles. When the key video frames in each video correspond to the same time, when acquiring the key video frames from the multiple videos from the multiple perspectives, the key video frames in one video can be used to determine the key video frames of each video, thereby improving data processing efficiency. Based on the object parameters of the target object in the key video frames of each video, a three-dimensional model of the target object at the key video frames is constructed. By using the object parameters in the key video frames from multiple perspectives, a three-dimensional model of the target object at the key video frames can be accurately constructed. Constructing a 3D model at each key video frame allows for the capture of dynamic changes in the target object over time, significantly reducing data redundancy and improving storage efficiency. Sending the 3D model of the target object at the key video frame, along with the location information of the key video frame and other video frames, to the client reduces the amount of data transmitted and lowers network bandwidth usage. Furthermore, the client can dynamically generate 3D models for other video frames based on the 3D model and location information at the key video frame, achieving efficient 4D scene reconstruction. Moreover, by arbitrarily setting virtual cameras in space, different perspectives of the target video can be flexibly generated to meet diverse needs, given the target video obtained from the 3D model.

[0057] See Figure 2 , Figure 2 A flowchart of a data processing method according to an embodiment of this specification is shown, which specifically includes the following steps.

[0058] Step 202: Obtain multiple videos of the target object from multiple perspectives, and obtain key video frames from each video from the multiple videos from multiple perspectives, wherein the multiple videos are time-synchronized and the key video frames of each video correspond to the same time.

[0059] The target object can be understood as a scene or object that can be captured from different perspectives. Specifically, the target object can be a dynamic scene (such as a person, a car, an animal, etc.) or a static scene (such as a building, a room, etc.). These scenes or objects may change over time (such as movement, deformation, etc.), and their complete information is captured through video from multiple perspectives.

[0060] Key video frames can be understood as representative key frames in a video frame sequence, which can be used to reduce the amount of data and as reference frames for subsequent processing.

[0061] Specifically, by acquiring videos of the target object from multiple different perspectives, the movement of the target object or changes in the scene are captured from different angles. Because the multiple videos are time-synchronized, the frames captured at the same moment are time-aligned. That is, when extracting key video frames from each video, the key video frames captured are also time-aligned; if a video frame is selected as the key video frame from the first video, then video frames at the same moment in other videos will also be selected as key video frames.

[0062] In one or more embodiments of this specification, image acquisition devices with different perspectives can simultaneously acquire multiple videos of a target object, thereby ensuring the acquisition of time-synchronized, multi-perspective videos of the target object. Specific implementation methods are described below:

[0063] The acquisition of multiple videos of the target object from multiple perspectives includes:

[0064] The system receives multiple videos of the target object sent by multiple image acquisition devices from different perspectives, wherein each video consists of multiple video frames.

[0065] Different perspectives can be understood as observing and capturing the target object from multiple angles or directions; each perspective provides different aspects of the target object, and this information will be fused in the subsequent 3D reconstruction to generate a complete 3D model of the target object.

[0066] Image acquisition devices can be understood as cameras or other video image acquisition devices (such as depth cameras, infrared cameras, etc.). These image acquisition devices are deployed at different locations around the target object to ensure that all-around information about the target object can be captured. Video is composed of a series of continuous still images (called video frames), each video frame representing the static state of the target object at a certain moment; each video consists of a large number of video frames, which are arranged in chronological order, recording the changes of the target object in the time dimension.

[0067] Specifically, multiple image acquisition devices start acquiring data at the same time, and their frame rates (the number of frames acquired per second) and timestamps are strictly aligned. Based on this synchronization, it can be ensured that videos from different perspectives are aligned in the time dimension, thereby enabling accurate reconstruction of the three-dimensional dynamic information of the target object in subsequent processing.

[0068] Each image acquisition device generates an independent video stream. These video streams are synchronized in time but have different viewpoints. Multiple videos provide dynamic information about the target object from different viewpoints.

[0069] The data processing method provided in the embodiments of this specification can obtain dynamic information of the target object from different perspectives by receiving multiple videos of the target object sent by multiple image acquisition devices from different perspectives, thus providing a data foundation for subsequent modeling.

[0070] In one or more embodiments of this specification, by selecting one video from multiple videos as an initial video, and by obtaining at least one key video frame from the initial video, a reference time corresponding to the at least one key video frame is determined, thereby efficiently obtaining the key video frames in each video based on the reference time. Specific implementation methods are described below:

[0071] The step of obtaining key video frames from multiple videos viewed from multiple perspectives includes:

[0072] An initial video is determined from multiple videos from the multiple perspectives, and at least one key video frame of the initial video is obtained;

[0073] Based on the reference time corresponding to the at least one key video frame, the video frame corresponding to the reference time in each video is determined as the key video frame.

[0074] The initial video can be understood as a video selected from multiple videos. This initial video can be used as a reference video for subsequent key video frame extraction. The selection of the initial video can be arbitrary and is not limited here. The reference time can be understood as the time point corresponding to the key video frame in the initial video. Since multiple videos are captured synchronously, these reference times are also aligned in other videos.

[0075] Specifically, in the initial video, multiple key video frames are extracted according to certain rules (such as the degree of scene change, time intervals, etc.). These key video frames represent important moments in the scene of the initial video. For example, in the initial video, video frames at t=1 second, t=3 seconds, and t=5 seconds are selected as key video frames.

[0076] Based on the reference time of the key video frame in the initial video, video frames at the same time are extracted from other videos as key video frames. For example, if the video frames at t=1 second, t=3 seconds, and t=5 seconds are key video frames, then the video frames at t=1 second, t=3 seconds, and t=5 seconds in other videos are also key video frames.

[0077] The data processing method provided in the embodiments of this specification uses an initial video selected from multiple videos as a reference video to determine the time points of key video frames. Key video frames of other videos are extracted synchronously based on these time points. The time synchronization of multiple videos ensures that the key video frames of multi-view videos are aligned in time. Therefore, key video frames of each video can be obtained efficiently based on the key video frames of a reference video.

[0078] In one or more embodiments of this specification, each video consists of multiple video frames. By analyzing the degree of variation between these multiple video frames, key video frames depicting scene changes are obtained from the multiple video frames. Specific implementation methods are described below:

[0079] Obtaining at least one key video frame of the initial video includes:

[0080] The at least one key video frame is determined from the plurality of video frames based on the degree of change between the plurality of video frames of the initial video.

[0081] The degree of change between multiple video frames can be understood as the difference in the geometric structure, appearance attributes, or scene content of the target object between adjacent video frames or video frames within a certain time interval. The degree of change can be quantified by calculating the pixel difference, feature point matching error, or other measurement methods between video frames. In the embodiments of this specification, the degree of change is an indicator of the deviation of the moving Gaussian point in the scene, measured by the Euclidean space length of the deviation, with the unit being millimeters.

[0082] By analyzing the degree of change between video frames, dynamic changes in the scene between video frames can be captured. Key video frames represent moments when significant changes occur in the scene. By extracting key video frames, the number of video frames that need to be processed can be reduced, thereby reducing the overhead of data storage and computation.

[0083] In practice, firstly, a current key video frame is selected from the initial video. Then, all video frames other than the current key video frame are sequentially designated as current video frames. By comparing the degree of change between the current key video frame and the current video frame, the current video frame can be updated to become the current key video frame. The specific implementation method is as follows:

[0084] The step of determining the multiple key video frames from the multiple video frames based on the degree of change between the multiple video frames of the initial video includes:

[0085] From the multiple video frames of the initial video, select and mark one video frame as the current key video frame, and then sequentially determine the other video frames from the multiple video frames as the current video frames;

[0086] The degree of change between the current video frame and the current key video frame is determined sequentially, and the current video frame whose degree of change exceeds a preset threshold is updated and marked as the current key video frame.

[0087] The multiple key video frames are obtained based on the multiple video frames that are marked as the current key video frames.

[0088] The preset threshold is used as a benchmark to determine whether the degree of change between scenes (between video frames) is significant. That is, if the degree of change exceeds the preset threshold, the degree of change between scenes is considered significant, and if the degree of change does not exceed the preset threshold, the degree of change between scenes is considered not significant. The preset threshold can be set according to the actual situation and is not limited here.

[0089] Specifically, from multiple video frames in the initial video, one video frame is selected as the current key video frame. For example, the video frame at t=0 seconds is selected as the current key video frame. In addition to the current key video frame, the other video frames in the initial video are sequentially marked as the current video frames.

[0090] These current video frames are compared sequentially with the current key video frame to determine the degree of change in the scene; for example, if the current key video frame is the video frame at t=0 seconds, then the video frames at t=1 seconds, t=2 seconds, t=3 seconds, etc., will be marked as the current video frames in sequence.

[0091] For each current video frame, calculate the degree of change between it and the current key video frame. If the degree of change between the current video frame and the current key video frame exceeds a preset threshold, it indicates a significant change in the scene, and the current video frame is updated to the new current key video frame. For example, if the degree of change between the current video frame at t=2 seconds and the current key video frame at t=0 seconds exceeds the preset threshold, then the current video frame at t=2 seconds is updated to the new current key video frame, and the degree of change between the current video frame at t=3 seconds and the current key video frame at t=2 seconds is calculated.

[0092] Through the above process, the frames in the video marked as the current key video frames constitute multiple key video frames. These key video frames represent moments in the scene where significant changes occur. For example, the final extracted key video frames may include video frames at t=0 seconds, t=2 seconds, t=5 seconds, etc.

[0093] The data processing method provided in the embodiments of this specification can automatically adjust the extraction frequency of key video frames according to the dynamic changes of the scene. That is, the extraction frequency of key video frames is higher in parts of the scene that change rapidly, and lower in parts of the scene that change slowly. This adaptability enables the method to efficiently handle various complex dynamic scenes. Moreover, the selection of key video frames is based on the degree of scene change, ensuring that the reconstructed 3D model can accurately reflect the dynamic changes of the scene, thereby improving the quality and realism of the 4D scene reconstruction.

[0094] In one or more embodiments of this specification, by selecting the first video frame in the initial video as the current key video frame, the degree of change between the determined current video frame and the first video frame is calculated, thereby reasonably determining each current key video frame in chronological order. Specific implementation methods are described below:

[0095] The step of selecting and marking a video frame as the current key video frame from multiple video frames of the initial video includes:

[0096] The first video frame among the multiple video frames of the initial video is determined as the current key video frame;

[0097] The step of updating and marking the current video frame whose change exceeds a preset threshold as the current key video frame includes:

[0098] If the degree of change exceeds the preset threshold, the current video frame is updated and marked as the current key video frame, and the steps of sequentially determining the degree of change between the current video frame and the current key video frame, and updating and marking the current video frame whose degree of change exceeds the preset threshold as the current key video frame are continued, until the degree of change between the current video frame and the current key video frame is determined.

[0099] In fact, if the degree of change does not exceed the preset threshold, the process of determining the degree of change between the current video frame and the current key video frame, and updating the current video frame whose degree of change exceeds the preset threshold as the current key video frame, continues until the degree of change between the current video frame and the current key video frame is determined.

[0100] Specifically, from the multiple video frames of the initial video, the first video frame (i.e. the first video frame) is selected as the current key video frame. For example, if the initial video starts at t=0 seconds, then the video frame at t=0 seconds is determined as the current key video frame.

[0101] For each current video frame, calculate the degree of change between it and the current key video frame. If the degree of change exceeds a preset threshold, update the current video frame as the new current key video frame. After updating the current key video frame, check if there is a next current video frame (i.e., a subsequent frame). If there is, continue with the following steps:

[0102] The next current video frame is compared with the new current key video frame to calculate the degree of change. If the degree of change exceeds the threshold, the current video frame is updated to the new current key video frame. If the degree of change does not exceed the threshold, the next current video frame is processed, and the above process is repeated until all current video frames have been processed (that is, the degree of change between each current video frame and the current key video frame has been determined).

[0103] For example, suppose the frame at t=1 second is updated to the new current key video frame. Next, the current video frame at t=2 second is processed. If the change between the current video frame at t=2 second and the current key video frame at t=1 second does not exceed a threshold, then the current video frame at t=3 second is processed. If the change between the current video frame at t=3 second and the current key video frame at t=1 second exceeds the threshold, then the current video frame at t=3 second is updated to the new current key video frame. This process continues until the last video frame of the initial video is processed.

[0104] The data processing method provided in the embodiments of this specification can capture moments of significant change in the scene by continuously updating the current key video frame when extracting key video frames from the initial video, thereby generating multiple key video frames. This method has the characteristics of adaptability, efficiency and high precision, and is particularly suitable for processing complex dynamic scenes.

[0105] Step 204: Based on the object parameters of the target object in the key video frames of each video, construct a three-dimensional model of the target object at the key video frame.

[0106] Specifically, object parameters can be understood as parameters that describe the geometric structure, appearance attributes, and other information of the target object. Object parameters include, but are not limited to, geometric parameters and appearance parameters. Geometric parameters can be understood as the three-dimensional shape, position, and pose of the target object. Appearance parameters can be understood as the color, texture, lighting, and other information of the target object.

[0107] In key video frames, the object parameters of the target object can be obtained through techniques such as feature point extraction, feature matching, and triangulation. For example, by extracting feature points in key video frames and using multi-view geometric methods (such as SFM, Structure from Motion, which is a technique for recovering three-dimensional structure and camera motion from two-dimensional image sequences), the three-dimensional position and pose of the target object can be calculated.

[0108] In fact, constructing a 3D model of the target object at the key video frame using the object parameters of the target object in the key video frame can be achieved in the following way:

[0109] Feature points of the target object are extracted from key video frames of each video, and feature point matching is performed between key video frames from multiple perspectives. The pose (rotation and translation) of each image acquisition device is estimated using the matched feature points. The positions of these feature points in 3D space are estimated using triangulation based on the camera pose and the matched feature points to obtain the corresponding 3D points. A set of Gaussian distribution parameters (mean and covariance matrix) is initialized based on the 3D points obtained by triangulation to represent the geometric structure and appearance attributes of the scene. The predicted Gaussian distribution parameters are reprojected onto a 2D image to obtain a projected 2D image. Image similarity loss is calculated between the projected 2D image and the actual 2D image, and the Gaussian distribution parameters are optimized based on the calculated loss value. The 3D model of the target object at the key video frames is obtained through 3D Gaussian Splatting.

[0110] For example, to obtain two-dimensional images from multiple perspectives (e.g., two images of the same cube taken by camera A and camera B), the 3D Gaussian distribution parameters can be initially estimated through triangulation (i.e., initializing a set of Gaussian distribution parameters, such as the 8 corner points of the cube, each corner point represented by a 3D Gaussian distribution parameter, including the mean position, covariance matrix, and color attribute). Assume the 3D Gaussian distribution parameters for one corner point are: mean (μ): 3D coordinates [1.0, 1.0, 2.0] (representing the corner point's position in space); covariance matrix (Σ): a 3x3 matrix describing the shape and direction of the Gaussian distribution (e.g., along the diagonal [0.1, 0.1, 0.1], representing an isotropic distribution); color (c): [255, 0, 0] (red).

[0111] The 3D Gaussian distribution parameters are projected onto the imaging plane of camera A. Assume the projected 2D Gaussian parameters are: μ_2d: [100,200] (pixel coordinates), Σ_2d: [[10,0],[0,10]] (circular distribution). In the 2D image, a red dot is rendered based on μ_2d and Σ_2d. In the actual image captured by camera A, the corner point is a clear red pixel (e.g., coordinates [105,205]). Mean squared error is used to compare the two image patches (e.g., a 5x5 pixel area centered at [100,200] and [105,205]) to obtain the loss value. Backpropagation is used to calculate the gradient of the loss value with respect to the 3D Gaussian distribution parameters, and the parameters are updated to optimize the 3D Gaussian distribution parameters.

[0112] By using key video frames from multiple perspectives and the object parameters within those frames, a 3D model of the target object can be accurately constructed, reflecting its geometric structure and appearance attributes at the key video frame moment. By fusing key video frame information from multiple perspectives, a complete 3D model of the target object can be generated, avoiding the limitations of single-perspective information. By constructing a 3D model at different key video frames, the dynamic changes of the target object in the time dimension can be captured, thereby achieving 4D (3D space plus time dimension) scene reconstruction.

[0113] In one or more embodiments of this specification, constructing a 3D model of the target object at the key video frame based on the object parameters of the target object in the key video frames of each video includes:

[0114] Based on the device parameters corresponding to the multiple image acquisition devices and the object parameters of the target object in the key video frames of each video, the Gaussian distribution parameters corresponding to the key video frames are obtained.

[0115] Based on the Gaussian distribution parameters corresponding to the key video frame, a three-dimensional Gaussian model of the target object at the key video frame is constructed.

[0116] Among them, device parameters refer to the intrinsic and extrinsic parameters of image acquisition devices (such as cameras). Intrinsic parameters include the focal length, principal point coordinates, distortion coefficients, etc. of the image acquisition device, which describe the internal optical characteristics of the camera. Extrinsic parameters include the position and orientation (rotation and translation) of the image acquisition device in the world coordinate system, which describe the external position and orientation of the camera. It should be noted that these device parameters are usually obtained through camera calibration (such as Zhang Zhengyou calibration method) or SFM (Structure from Motion) technology.

[0117] Gaussian distribution parameters refer to the Gaussian distribution parameters used to describe the geometric structure and appearance attributes of a target object. They typically include the mean (μ) and the covariance matrix (Σ). The mean is used to represent the center position of the Gaussian distribution, describing the three-dimensional position of the target object, while the covariance matrix is ​​used to represent the shape and orientation of the Gaussian distribution, describing the geometric structure of the target object (such as size, shape, and orientation).

[0118] The process of obtaining Gaussian distribution parameters can be found in the above embodiments, and will not be repeated here.

[0119] In fact, a three-dimensional model of a target object can be obtained through techniques such as point cloud reconstruction and mesh reconstruction. Specifically, point cloud reconstruction uses the center point (mean) of a Gaussian distribution as a three-dimensional point cloud to represent the geometric structure of the target object; mesh reconstruction converts the Gaussian distribution into a mesh model (such as a triangular mesh) to more intuitively represent the shape of the target object.

[0120] In the embodiments of this specification, the three-dimensional Gaussian model can be understood as a probability distribution-based representation method. It uses a three-dimensional Gaussian distribution (multivariate normal distribution) to describe the data. Specifically, it uses a set of Gaussian distribution parameters to describe the geometric structure and appearance attributes of the target object. A small number of Gaussian distributions can replace a large number of original data points (such as point clouds), thereby reducing storage and computational requirements. Furthermore, by adjusting the Gaussian distribution parameters, efficient and high-quality rendering of complex scenes can be achieved, such as 3D Gaussian Splatting.

[0121] One key advantage of using Gaussian distribution parameters for rendering is that it can significantly reduce the amount of data that needs to be processed. Compared to directly processing large amounts of 3D point cloud or mesh data, by approximating the points in the scene with a small but carefully selected Gaussian distribution, rendering efficiency can be significantly improved while maintaining visual quality.

[0122] The data processing method provided in the embodiments of this specification can accurately describe the geometric structure and appearance attributes of the target object through Gaussian distribution parameters, thereby achieving high-precision 3D reconstruction. Gaussian distribution parameters can be used for efficient 3D rendering techniques (such as 3D Gaussian Splatting) and reduce the amount of data to be stored and computed.

[0123] In one or more embodiments of this specification, before obtaining Gaussian distribution parameters and constructing a 3D model, distortion correction processing is performed on multiple acquired videos to eliminate geometric distortion caused by the lens of the image acquisition device and improve the accuracy of the reconstructed 3D model. Specific implementation methods are described below:

[0124] The step of obtaining the Gaussian distribution parameters corresponding to the key video frames based on the device parameters corresponding to the multiple image acquisition devices and the object parameters of the target object in the key video frames of each video includes:

[0125] Using the multiple videos, obtain the device parameters corresponding to the multiple image acquisition devices;

[0126] Based on the distortion coefficients in the device parameters corresponding to the multiple image acquisition devices, distortion removal processing is performed on the video corresponding to the image acquisition devices to obtain multiple processed videos.

[0127] Based on the intrinsic and extrinsic parameters of the corresponding device parameters of each image acquisition device and the object parameters of the target object in the key video frames of the processed multiple videos, the Gaussian distribution parameters corresponding to the key video frames are obtained.

[0128] Specifically, obtaining the device parameters corresponding to the multiple image acquisition devices using the multiple videos includes:

[0129] Based on the multiple videos and the multiple video frames corresponding to the target time, the multiple image acquisition devices are calibrated to obtain the device parameters corresponding to the multiple image acquisition devices, wherein the target time is any one of the multiple times in the video.

[0130] Specifically, by acquiring multiple videos, a time series containing multiple moments can be determined for each video. By randomly selecting a target moment from the multiple moments, multiple video frames corresponding to the target moment can be obtained. Multiple image acquisition devices can be calibrated using multiple video frames from the same moment but different perspectives. In practice, feature points can be extracted from multiple video frames, and feature point matching can be performed between video frames from multiple perspectives. Using the matched feature points, the intrinsic and extrinsic parameters of each image acquisition device can be estimated using SFM technology. During the calibration process, the distortion coefficients (such as radial distortion, tangential distortion, etc.) of the image acquisition devices are also estimated. The distortion coefficients describe the image distortion introduced by the lens of the image acquisition device.

[0131] To eliminate these distortions, the video needs to be processed for distortion correction. After distortion correction, a processed video is generated. The video frames in this processed video are no longer affected by lens distortion, making them more suitable for subsequent 3D reconstruction and ensuring the accuracy of the subsequent reconstruction process.

[0132] In practical applications, the distortion coefficients of image acquisition devices are used to perform distortion correction processing on the videos acquired by those devices. For example, if a first video is acquired using a first image acquisition device and a second video is acquired using a second image acquisition device, the first video is distorted using the first distortion coefficient corresponding to the first image acquisition device, and the second video is distorted using the second distortion coefficient corresponding to the second image acquisition device.

[0133] Using the distortion-corrected video, key video frames of multiple processed videos are obtained. Based on the intrinsic and extrinsic parameters of each image acquisition device and the object parameters of the target object in the key video frames, the corresponding Gaussian distribution parameters at the key video frames are obtained.

[0134] The specific implementation method is similar to the method described above for obtaining the Gaussian distribution parameters corresponding to key video frames using multiple videos before distortion correction, and will not be repeated here.

[0135] The data processing method provided in the embodiments of this specification generates an optimized video through distortion removal processing, which eliminates the impact of lens distortion on image quality and improves the accuracy of 3D reconstruction. Based on the optimized video and device parameters, the Gaussian distribution parameters at key video frames can be accurately obtained, thereby achieving high-precision 3D reconstruction and dynamic scene modeling.

[0136] Step 206: Send the 3D model of the target object at the key video frame and the position information of the key video frame and other video frames to the client, wherein the other video frames are the video frames other than the key video frame in each video. The 3D model and the position information are used to enable the client to obtain the 3D model of the target object at the other video frames, and to obtain the target video based on the 3D model of the target object at the key video frame and the 3D model of the target object at the other video frames.

[0137] Location information can be understood as the position (timestamp) of the key video frame and other video frames on the timeline, as well as their relative positional relationships (such as inter-frame offset). For example, the key video frame is located at t=1 second, while other video frames may be located at t=1.1 seconds, t=1.2 seconds, etc.; location information can be used to describe the temporal relationships and dynamic changes between these frames.

[0138] Specifically, the 3D model of the target object at the key video frame, as well as the positional information between the key frame and other video frames, is sent to the client. After receiving the 3D model and positional information of the key video frame, the client can generate the 3D model of the target object at other video frames through interpolation or prediction methods.

[0139] In a specific embodiment, the client can generate a 3D model of other video frames using a prediction algorithm (such as based on a Transformer model); or generate a 3D model of intermediate frames using an interpolation algorithm based on the 3D model of the key frame and the position information of other video frames. For example, if the key video frames are located at t=1 second and t=3 seconds, the client can generate a 3D model of the corresponding video frame at t=2 seconds by interpolation.

[0140] After obtaining the 3D model of the target object in key video frames and other video frames, the client can generate the target video using 3D rendering technology. Specifically, the 3D model of the target object in each video frame is rendered as a 2D image, and the rendered 2D images are combined in chronological order to generate the target video.

[0141] By sending the 3D model and location information of key video frames, the amount of data that needs to be transmitted is reduced, thus lowering the network bandwidth usage. On the client side, the 3D model of other video frames can be dynamically generated based on the 3D model and location information of the key video frames, thereby achieving efficient 4D scene reconstruction.

[0142] In practical applications, during stage performances, multiple cameras with different perspectives can be used to capture images of the stage scene (i.e., the target object in the above embodiments), obtaining multiple art-related videos from multiple perspectives. From these multiple art-related videos, key video frames are obtained for each art-related video. These multiple art-related videos are time-synchronized, and the key video frames correspond to the same time. Based on the scene parameters of the stage scene (i.e., the object parameters in the above embodiments) in the key video frames of each art-related video, a 3D model of the stage scene at the key video frame is constructed. The 3D model of the stage scene at the key video frame and the key video frame are then compared. The positional information between the key video frame and other video frames is sent to the client. The other video frames are the video frames in each art category other than the key video frame. The 3D model and the positional information are used to enable the client to obtain the 3D model of the stage scene at the other video frames. Through the 3D model of the stage scene at the key video frame and other video frames, a 4D model of the stage scene (3D model plus time dimension) can be reconstructed. Based on this, the 3D model of the stage scene can be rendered at any viewpoint and any time point as needed to generate a target video. For example, a video showing a 360-degree stage scene that changes rhythm according to the stage performance can be generated.

[0143] In another embodiment, in a sports broadcasting scenario, multiple cameras with different perspectives can be used to capture images of the sports stadium (i.e., the target object in the above embodiment), obtaining multiple sports videos from multiple perspectives. Similarly, key video frames of each sports video are obtained from the multiple sports videos from multiple perspectives, wherein the multiple sports videos are time-synchronized, and the key video frames of each sports video correspond to the same time. Based on the scene parameters of the sports stadium (i.e., the object parameters in the above embodiment) in the key video frames of each sports video, a 3D model of the sports stadium at the key video frames is constructed. The 3D model of the sports stadium at the key video frames and... The positional information between the key video frame and other video frames is sent to the client. The other video frames are the video frames in each sports video category other than the key video frame. The 3D model and the positional information are used to enable the client to obtain the 3D model of the sports stadium at the other video frames. Through the 3D model of the sports stadium at the key video frame and other video frames, a 4D model of the sports stadium (3D model plus time dimension) can be reconstructed. Similarly, the 3D model of the sports stadium can be rendered at any angle and at any time point as needed to generate the target video. For example, it can provide users with a multi-angle viewing experience or display exciting clips in 360 degrees.

[0144] Of course, this data processing method can also be applied to educational scenarios. In this case, the multiple videos obtained from multiple perspectives can be educational videos, such as those in courses requiring fine motor skills (e.g., medical surgery instruction). Multiple cameras with different perspectives can be used to capture images of the operating table (i.e., the target object in the above embodiment), obtaining multiple educational videos from multiple perspectives. From these multiple educational videos, key video frames are obtained for each video. These multiple educational videos are time-synchronized, and the key video frames of each video correspond to the same time. Based on the scene parameters of the operating table (i.e., the object parameters in the above embodiment) in the key video frames of each educational video, the operation... The system generates a 3D model of the console at the key video frame. The 3D model of the console at the key video frame, along with the positional information between the key video frame and other video frames, is sent to the client. These other video frames are the video frames in each educational video category other than the key video frame. The 3D model and positional information are used by the client to obtain the 3D model of the console at the other video frames. Using the 3D models of the console at the key video frame and other video frames, a 4D model of the console (3D model plus time dimension) can be reconstructed. Similarly, the 3D model of the console can be rendered from any viewpoint and at any time point as needed to generate the target video, helping students observe and learn from different angles.

[0145] The data processing method provided in one embodiment of this specification can reselect key video frames through dynamic scene change statistics, achieving dynamic adaptive adjustment of key video frames. Furthermore, at key video frames, the complete 3D model of the scene is reconstructed, while at other non-key frames, complex scene reconstruction is unnecessary, thereby improving data processing efficiency. In this case, storing the 3D model at the key video frames can also greatly reduce data redundancy and improve storage efficiency. Since the 3D models at other video frames can be predicted using the 3D models at the key video frames, data compression is achieved. Simultaneously, based on 3D Gaussian Splatting technology, it provides an efficient and accurate solution for four-dimensional dynamic reconstruction. The key lies in combining adaptive methods and compressed reconstruction strategies to achieve fast rendering without sacrificing accuracy, thus balancing real-time performance and accuracy, and maintaining a balance between high computational performance and reconstruction accuracy.

[0146] See Figure 3 , Figure 3 A flowchart illustrating a data processing method applied to a client according to an embodiment of this specification is shown, specifically including the following steps.

[0147] Step 302: Receive the 3D model of the target object at the key video frame, as well as the position information of the key video frame and other video frames, wherein the key video frame is the key video frame of the target object in multiple videos from multiple perspectives, and the other video frames are the video frames in the multiple videos other than the key video frame.

[0148] Step 304: Input the 3D model at the key video frame and the location information into the target parameter prediction network to obtain the 3D model of the target object at the other video frames.

[0149] Step 306: Obtain the target video based on the 3D model of the target object at the key video frame and the 3D model at the other video frames.

[0150] The target parameter prediction network can be understood as a machine learning model used to predict the 3D model of a target object in other video frames. For example, a Transformer model can predict the 3D model of a target object in other frames by capturing the relative position information between keyframes and other video frames.

[0151] The input data of the target parameter prediction network includes a 3D model at the key video frame (recording the geometric structure and appearance attributes of the target object at the key video frame) and location information (recording the timestamps of the key frame and other video frames and their relative positional relationship); the output data is the 3D model at other video frames, that is, the geometric structure and appearance attributes of the target object at other video frames.

[0152] The target parameter prediction network can capture the relative positional relationship between key video frames and other video frames by using the positional information (such as timestamps and inter-frame offsets) of key video frames and other video frames. In the embodiments of this specification, the positional information between key video frames and other video frames can be understood as the inter-frame offset between them. This offset is calculated using the sequence numbers of other video frames and the key video frame sequence number, where the video frame sequence number represents the order information of the frame in the video sequence. To more effectively model the positional relationship between key video frames and other video frames, position embedding technology can be used to map the inter-frame offset into a high-dimensional vector representation, thereby enhancing the network's ability to model temporal relationships.

[0153] The target parameter prediction network predicts the 3D model of the target object in other video frames based on the 3D model and location information of key video frames.

[0154] Specifically, the target parameter prediction network is trained through the following steps:

[0155] Obtain multiple video samples of the object sample from multiple perspectives, and construct a three-dimensional model sample of the object sample at each video frame sample based on the object parameters of the object sample in the video frame sample of each video sample.

[0156] From the video frame samples of each video sample, determine the key video frame samples and other video frame samples of each video sample;

[0157] The Gaussian distribution parameters corresponding to the key video frame samples and the 3D model samples, as well as the position information of the key video frame samples and other video frame samples, are input into the initial parameter prediction network to obtain the predicted Gaussian distribution parameters of the other video frame samples.

[0158] Based on the Gaussian distribution parameters corresponding to the other video frame samples and the 3D model samples, as well as the predicted Gaussian distribution parameters, the initial parameter prediction network is trained to obtain the target parameter prediction network.

[0159] Here, object samples can be understood as target objects in the training data, which can be dynamic scenes (such as people or animals) or static scenes (such as buildings). Multiple video samples from multiple perspectives can be understood as video data of object samples captured synchronously from different angles by multiple image acquisition devices, with each video sample providing dynamic information of the object sample from different perspectives.

[0160] 3D model samples can be understood as 3D models of object samples constructed at various video frame samples through multi-view feature extraction, camera pose estimation, triangulation, and other techniques. These 3D models are typically represented by Gaussian distribution parameters (mean and covariance matrix). Key video frame samples can be understood as representative frames in the video samples, usually selected based on the degree of scene variation. Other video frame samples can be understood as frames other than key video frames, typically used to train the model to predict 3D models of non-key video frames.

[0161] An initial parameter prediction network can be understood as a model to be trained, used to predict the Gaussian distribution parameters of a target object at other video frame samples. For example, an initial parameter prediction network is a Transformer structure model. Predicting Gaussian distribution parameters can be understood as the initial parameter prediction network predicting the Gaussian distribution parameters of other video frame samples based on the Gaussian distribution parameters and location information at the input key video frame samples.

[0162] The Gaussian distribution parameters at other video frame samples can be understood as the true Gaussian distribution parameters of the target object at other video frame samples, obtained through 3D model samples. Predicting the Gaussian distribution parameters can be understood as the prediction results generated by the initial parameter prediction network.

[0163] By comparing the predicted Gaussian distribution parameters with the true Gaussian distribution parameters, a loss function is calculated. Commonly used loss functions include mean squared error (MSE) or KL divergence. The parameters of the initial parameter prediction network are adjusted using the backpropagation algorithm to minimize the loss function. This process is repeated until the model converges, ultimately yielding the target parameter prediction network. In other words, through multiple iterations of training, the initial parameter prediction network gradually learns how to accurately predict the Gaussian distribution parameters of other video frame samples based on the Gaussian distribution parameters and location information of key video frame samples.

[0164] The data processing method provided in the embodiments of this specification obtains a target parameter prediction network through training, which can accurately predict the Gaussian distribution parameters of the target object at other video frames, thereby achieving high-precision 3D model generation. Moreover, the trained target parameter prediction network can generalize to new video data and is applicable to various scenarios and target objects.

[0165] In one or more embodiments of this specification, the Gaussian distribution parameters of the target object at other video frames are predicted by a target parameter prediction network, and a three-dimensional model of the target object at other video frames is constructed using the Gaussian parameters at other video frames.

[0166] The step of inputting the 3D model at the key video frame and the location information into the target parameter prediction network to obtain the 3D model of the target object at other video frames includes:

[0167] The Gaussian distribution parameters corresponding to the 3D model at the key video frame and the location information are input into the target parameter prediction network to obtain the Gaussian distribution parameters of the target object at other video frames.

[0168] Based on the Gaussian distribution parameters of the target object in other video frames, a three-dimensional Gaussian model of the target object in other video frames is obtained.

[0169] Specifically, the 3D model at the key video frame and the positional information between the key video frame and other video frames are input into the target parameter prediction network obtained through training. The target parameter prediction network obtains the Gaussian distribution parameters of the target object at the other video frames, thereby obtaining the 3D model of the target object at the other video frames based on the Gaussian distribution parameters of the target object at the other video frames.

[0170] For detailed implementation methods, please refer to the above embodiments, which will not be repeated here.

[0171] The data processing method provided in the embodiments of this specification greatly reduces data redundancy by combining Transformer and key video frames, and can accurately describe the geometric structure and appearance attributes of the target object through Gaussian distribution parameters, thereby achieving high-precision three-dimensional reconstruction.

[0172] In one or more embodiments of this specification, by obtaining a three-dimensional model of the target object at each moment and adding a time dimension, a four-dimensional volumetric video of the target object is obtained. This four-dimensional volumetric video can be played continuously. By arbitrarily setting up a virtual camera in space, the 3D model at that moment is rendered onto a 2D image, thus obtaining a target video. Specific implementation methods are described below:

[0173] The step of obtaining the target video based on the 3D model of the target object at the key video frame and the 3D model at other video frames includes:

[0174] Based on the 3D model of the target object at the key video frame and the 3D model at other video frames, obtain the 3D model of the target object at each video frame;

[0175] Based on the 3D model of the target object at each video frame and the time information, the volumetric video corresponding to the target object is obtained, wherein the time information is used to represent the timestamp of the video frame;

[0176] The video of the specified volume is rendered to obtain the target video.

[0177] The time information can be understood as the timestamp corresponding to each frame, used to indicate the position of the target object on the timeline. Volumetric video can be understood as a 4D volumetric video (including three-dimensional space and time dimensions), obtained by combining the three-dimensional model of the target object at each moment in chronological order; target video can be understood as the final rendered video.

[0178] Specifically, by combining the 3D models of key video frames and other video frames, a 3D model of the target object at each video frame can be obtained. The 3D models of the target object at each video frame are combined in chronological order to form a 4D volumetric video. This volumetric video can be regarded as a time-series 3D model that describes the dynamic changes of the target object in the time dimension. That is, the 4D volumetric video not only shows the changes of the target object over time (the fourth dimension), but also contains comprehensive information such as its shape, position, and posture.

[0179] The 3D model at each video frame in the volumetric video is rendered as a 2D image. Specifically, the commonly used 3D Gaussian rendering technique can be used to render the reconstructed 3D model as a 2D image. The rendered 2D images are then combined in chronological order to generate the target video. That is, the target video is a coherent 2D video sequence that shows the dynamic changes of the target object in the time dimension.

[0180] Based on multi-view video, the motion in the scene is fully replicated through 3D modeling to obtain a virtual 3D scene. Adding the time dimension, a 4D volumetric video is obtained. This volumetric video can be played continuously. In space, a virtual camera can be set arbitrarily to render the 3D scene at that moment onto a 2D image. The rendered 2D images are combined in chronological order to generate the target video.

[0181] The data processing method provided in the embodiments of this specification uses 3D Gaussian rendering technology to render the reconstructed three-dimensional model into a 2D image and obtain the target video according to the time sequence. The target video generated by 3D Gaussian rendering technology has high precision and high realism, can accurately reflect the dynamic changes of the target object, and can render the three-dimensional model of the target object from any viewpoint and at any time point as needed, thereby generating diverse target videos.

[0182] The data processing method provided in the embodiments of this specification is based on a target parameter prediction network obtained through training. It maps the Gaussian distribution parameters of key video frames to other video frames currently being processed, obtains the Gaussian distribution parameters of other video frames, and then renders the reconstructed three-dimensional model into a 2D image through 3D Gaussian rendering technology to obtain the target video. The generated target video has high precision and high realism, takes into account both real-time performance and accuracy, and maintains a balance between efficient computing performance and reconstruction accuracy.

[0183] The following is in conjunction with the appendix Figures 4a-4c The data processing method will be further explained below. Figure 4a This specification illustrates a flowchart of a data preprocessing process in a data processing method according to an embodiment of the present specification, which specifically includes the following steps.

[0184] The data preprocessing section is mainly used to obtain time-synchronized videos of the target object from multiple perspectives and to perform distortion correction on the videos.

[0185] Step 402: Use a multi-view camera to acquire data synchronously.

[0186] Specifically, the hardware system for data acquisition consists of multiple synchronized cameras with different viewpoints (i.e., the image acquisition devices in the above embodiments). The cameras record time-synchronized video, which serves as the input to the software system. For example, 10 cameras record simultaneously, with each camera recording 50 frames per second.

[0187] Synchronous acquisition by multi-view cameras ensures that video data with consistent time is obtained from different perspectives. Through multi-view acquisition, dynamic changes in the scene can be captured more accurately.

[0188] Step 404: Extract a single frame from the video.

[0189] Extracting each frame from the acquired video in chronological order, thus obtaining each video frame, helps to segment continuous dynamic video information into processable static frames, facilitating subsequent processing steps.

[0190] Step 406: SFM camera calibration.

[0191] Camera calibration is performed on each frame using Structure from Motion (SFM) technology to obtain the intrinsic and extrinsic parameters of each camera. This step ensures the accuracy of subsequent reconstruction processes. The calibration incorporates the actual physical world scale by inputting the actual distance from the camera to the target location.

[0192] In fact, by obtaining a set of multiple synchronized video frames from multiple perspectives at the same time, camera calibration can be performed to obtain the intrinsic and extrinsic parameters of each camera. It should be noted that the intrinsic parameters of each camera include the distortion coefficient.

[0193] Step 408: Dedistort the video from each viewpoint.

[0194] By using the distortion coefficients of each camera, the video from the corresponding camera viewpoint is processed to remove distortion, eliminating the geometric distortion caused by the camera lens, thereby improving the reconstruction accuracy.

[0195] Figure 4b A flowchart illustrating the dynamic reconstruction process in a data processing method provided in one embodiment of this specification is shown.

[0196] After distortion correction is performed on each video frame, 3D reconstruction of the captured scene can be performed.

[0197] Step 410: Select the initial frame as the keyframe.

[0198] In the embodiments of this specification, keyframes (i.e., key video frames in the above embodiments) can be adaptively selected and updated. Initially, the first frame of each video is selected as the keyframe, and subsequent keyframes can be updated based on the degree of change between frames.

[0199] Step 412: Initialize the dynamic state to 0.

[0200] The dynamic state (i.e., the degree of change in the above embodiments) is an index of the deviation of moving Gaussian points in the scene, measured in Euclidean space length in millimeters; it is used to calculate the difference between the current frame (i.e., the current video frame in the above embodiments) and the key frame, thereby quantifying the dynamic changes of the scene.

[0201] Step 414: Compressed Reconstruction.

[0202] At keyframes, a complete 3D Gaussian model of the scene is reconstructed. This 3D Gaussian model describes the geometric structure and appearance attributes of the scene through a set of Gaussian distribution parameters. By replacing a large number of original data points (such as point clouds) with a small number of Gaussian distributions, storage and computational requirements can be reduced. At non-keyframes, it is not necessary to completely reconstruct the 3D Gaussian model of the scene. Subsequently, a time-aware lightweight transformer network can be used to map the Gaussian distribution parameters of the keyframes to the current frame to obtain the Gaussian distribution parameters of the current frame.

[0203] Step 416: Reset keyframes.

[0204] When the dynamic state exceeds the state threshold (i.e., the preset threshold in the above embodiment), the current frame is reset as a key frame. Previous key frames can be stored in the historical key frame list, and the relevant information of the key frames is recorded. This relevant information includes the position of the key frame (which can be represented by the sequence number of the key frame in the video frame sequence) and the segment length between the key frame and the next key frame (referring to the time distance between two adjacent key frames).

[0205] When the proportion of moving parts in the scene is high, keyframes will be reset more quickly. When the proportion of moving parts in the scene is low, keyframes will be reused more to improve system storage and computing efficiency.

[0206] Figure 4c A flowchart illustrating the dynamic rendering process in a data processing method provided in one embodiment of this specification is shown.

[0207] Given the 3D Gaussian model of the keyframe, the 3D Gaussian model of the current frame can be generated using the 3D Gaussian model of the keyframe, the positional information between the current frame and the keyframe, and the Transformer.

[0208] Step 418: Based on keyframe information, location information, and the Transformer network, obtain the Gaussian distribution parameters of the current frame.

[0209] The position information refers to the positional deviation between the current frame and the key frame, which can be obtained by calculating the difference between the current frame number and the key frame number.

[0210] Specifically, keyframe information includes the Gaussian distribution parameters at the keyframe, as well as the aforementioned related information (the position of the keyframe, the segment length between this keyframe and the next keyframe). By using the position and segment length of the keyframe, temporal modeling and positional information encoding can be enhanced. Thus, during the decoding process, when the Gaussian distribution parameters at the keyframe are mapped to the current frame using the Transformer network, a more accurate Gaussian distribution parameter for the current frame can be obtained.

[0211] In fact, at non-keyframe locations, position embedding based on the deviation between the current frame and the keyframe can optimize a time-aware lightweight transformer network. In practice, by introducing the deviation between the current frame and the keyframe to construct position embedding, temporal relationships can be captured, enhancing the network's ability to model temporal relationships.

[0212] The following is the training process for the Transformer network:

[0213] Input data: Position embedding of the deviation between the current frame and the keyframe, and Gaussian distribution parameters of the scene at the keyframe. Output data: Predicted Gaussian distribution parameters for the current frame.

[0214] The predicted Gaussian distribution parameters are compared with the true Gaussian distribution parameters of the current frame (obtained by reconstructing the 3D Gaussian model at the current frame), the loss function is calculated, and the network parameters of the Tranformer network are adjusted.

[0215] Therefore, when applying the Transformer network, the Gaussian distribution parameters of the current frame are obtained by taking the positionembedding based on the deviation between the current frame and the key frame, and the Gaussian distribution parameters of the scene at the key frame as input.

[0216] Based on this, the Transformer network can be used to obtain the Gaussian distribution parameters corresponding to each video frame, that is, to obtain the three-dimensional Gaussian model at each video frame.

[0217] Step 420: Render the reconstructed 3D model into a 2D image using 3D Gaussian rendering technology.

[0218] By using 3D Gaussian rendering technology, the reconstructed 3D model is rendered into a 2D image, which allows for the creation of dynamically changing volumetric videos from multi-view video data and their realistic presentation on a 2D screen.

[0219] Specifically, by using multi-view video, all the motion in the scene is replicated to obtain a virtual 3D scene. Adding the time dimension, a 4D volumetric video is obtained. Rendering is the process of continuously playing this 4D volumetric video. Virtual cameras can be arbitrarily set in space to render the 3D scene at that moment onto a 2D image to obtain a target video.

[0220] Of course, multiple perspectives of video over the entire duration can also be represented in the same 4D Gaussian splatting model, which introduces the time dimension into the traditional 3D Gaussian splatting technique. This method can simultaneously model spatial (3D) and temporal (1D) information, thereby achieving efficient representation and rendering of dynamic scenes. In the embodiments of this specification, 3D Gaussian splatting uses three-dimensional Gaussian distribution parameters to represent points or regions in the scene. Each Gaussian distribution parameter is defined by the mean (position) and covariance matrix (shape and orientation). 4D Gaussian splatting, on the other hand, introduces a time dimension on top of the three dimensions, extending the Gaussian distribution parameters to four dimensions, that is, adding time information on top of the mean and covariance matrix.

[0221] The data processing method provided in the embodiments of this specification offers an efficient and accurate solution for 4D dynamic reconstruction using Gaussian Splatting technology. Its key lies in combining adaptive methods and compression reconstruction strategies to achieve rapid rendering without sacrificing accuracy. Specifically, scene change statistics enable adaptive adjustment of dynamic keyframes, and the combination of Transformer networks and 3D Gaussian splashing technology achieves efficient compression. This data processing method, through innovative adaptive technology and compression reconstruction strategies, achieves efficient processing of long sequences of 4D scenes, possessing broad application prospects and practical value, especially providing an excellent solution for scenarios requiring high dynamics and real-time display.

[0222] Corresponding to the above method embodiments, this specification also provides embodiments of a data processing system. Figure 5 A schematic diagram of the structure of a data processing system according to one embodiment of this specification is shown. Figure 5 As shown, the system includes:

[0223] The server 502 is configured to acquire multiple videos of a target object from multiple perspectives, and to acquire key video frames from each video from the multiple videos from multiple perspectives, wherein the multiple videos are time-synchronized and the key video frames of each video correspond to the same time; to construct a three-dimensional model of the target object at the key video frame based on the object parameters of the target object in the key video frame of each video; and to send the three-dimensional model of the target object at the key video frame and the position information between the key video frame and other video frames to the client.

[0224] The client 504 is configured to receive a 3D model of the target object at a key video frame and position information between the key video frame and other video frames; input the 3D model at the key video frame and the position information into a target parameter prediction network to obtain a 3D model of the target object at other video frames; and obtain a target video based on the 3D model of the target object at the key video frame and the 3D model at other video frames.

[0225] Optionally, the server 502 is further configured to:

[0226] The system receives multiple videos of the target object sent by multiple image acquisition devices from different perspectives, wherein each video consists of multiple video frames.

[0227] Optionally, the server 502 is further configured to:

[0228] An initial video is determined from multiple videos from multiple perspectives, and at least one key video frame of the initial video is obtained; based on the reference time corresponding to the at least one key video frame, the video frame corresponding to the reference time in each video is determined as the key video frame.

[0229] Optionally, the server 502 is further configured to:

[0230] The at least one key video frame is determined from the plurality of video frames based on the degree of change between the plurality of video frames of the initial video.

[0231] Optionally, the server 502 is further configured to:

[0232] From the multiple video frames of the initial video, select and mark one video frame as the current key video frame, and then sequentially determine the other video frames from the multiple video frames as current video frames; sequentially determine the degree of change between the current video frame and the current key video frame, and update the marking of the current video frame whose degree of change exceeds a preset threshold as the current key video frame; obtain the multiple key video frames based on the multiple video frames marked as the current key video frames.

[0233] Optionally, the server 502 is further configured to:

[0234] The first video frame among the multiple video frames of the initial video is identified as the current key video frame. If the degree of change exceeds the preset threshold, the current video frame is updated and marked as the current key video frame, and the steps of sequentially determining the degree of change between the current video frame and the current key video frame, and updating and marking the current video frame whose degree of change exceeds the preset threshold as the current key video frame are continued, until the degree of change between the current video frame and the current key video frame is determined.

[0235] Optionally, the server 502 is further configured to:

[0236] Based on the device parameters corresponding to the multiple image acquisition devices and the object parameters of the target object in the key video frames of each video, the Gaussian distribution parameters corresponding to the key video frames are obtained; based on the Gaussian distribution parameters corresponding to the key video frames, a three-dimensional Gaussian model of the target object at the key video frames is constructed.

[0237] Optionally, the server 502 is further configured to:

[0238] Using the multiple videos, obtain the device parameters corresponding to the multiple image acquisition devices; perform distortion correction processing on the videos corresponding to the image acquisition devices based on the distortion coefficients in the device parameters corresponding to the multiple image acquisition devices to obtain multiple processed videos; obtain the Gaussian distribution parameters corresponding to the key video frames based on the intrinsic and extrinsic parameters of the device parameters corresponding to each image acquisition device and the object parameters of the target object in the key video frames of the multiple processed videos.

[0239] Optionally, the client 504 is further configured to:

[0240] The Gaussian distribution parameters corresponding to the 3D model at the key video frame and the location information are input into the target parameter prediction network to obtain the Gaussian distribution parameters of the target object at other video frames; based on the Gaussian distribution parameters of the target object at other video frames, the 3D Gaussian model of the target object at other video frames is obtained.

[0241] Optionally, the client 504 is further configured to:

[0242] Based on the 3D model of the target object at the key video frame and the 3D model at other video frames, a 3D model of the target object at each video frame is obtained; based on the 3D model of the target object at each video frame and time information, a volumetric video corresponding to the target object is obtained; the volumetric video is rendered to obtain the target video, wherein the time information is used to represent the timestamp of the video frame.

[0243] This specification provides a data processing system in one embodiment. The server acquires multiple videos of a target object from multiple perspectives, capturing the movement or scene changes of the target object from different angles. When the key video frames in each video correspond to the same time, when acquiring the key video frames from the multiple videos from the multiple perspectives, the key video frames in one video can be used to determine the key video frames of each video, thereby improving data processing efficiency. Based on the object parameters of the target object in the key video frames of each video, a three-dimensional model of the target object at the key video frames is constructed. By using the object parameters in the key video frames from multiple perspectives, a three-dimensional model of the target object at the key video frames can be accurately constructed. Constructing a 3D model at each key video frame allows for the capture of dynamic changes in the target object over time, significantly reducing data redundancy and improving storage efficiency. Sending the 3D model of the target object at the key video frame, along with the location information of the key video frame and other video frames, to the client reduces the amount of data transmitted and lowers network bandwidth usage. Furthermore, the client can dynamically generate 3D models for other video frames based on the 3D model and location information at the key video frame, achieving efficient 4D scene reconstruction. Moreover, by arbitrarily setting virtual cameras in space, different perspectives of the target video can be flexibly generated to meet diverse needs, given the target video obtained from the 3D model.

[0244] The above is an illustrative scheme of a data processing system according to this embodiment. It should be noted that the technical solution of this data processing system and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the data processing system, please refer to the description of the technical solution of the data processing method described above.

[0245] Figure 6 A structural block diagram of a computing device 600 provided according to one embodiment of this specification is shown.

[0246] The computing device 600 includes:

[0247] Memory 610 and processor 620;

[0248] The memory 610 is used to store computer programs / instructions, and the processor 620 is used to execute the computer programs / instructions, which, when executed by the processor 620, implement the steps of the data processing method.

[0249] In one or more embodiments of this specification, the computing device can be understood as an integrated smart terminal, including but not limited to a server, desktop computer, PC (Personal Computer), all-in-one model machine, mobile phone, tablet computer or other portable smart terminal, etc., and the computing device may have the model described in the above embodiments of this application pre-installed.

[0250] Specifically, this computing device can pre-install various types of models, including but not limited to models in natural language processing, visual processing, speech processing, code processing, and multimodal task processing, thus providing diverse model selection. In different product forms, this computing device can support one or more model usage methods, including but not limited to model training, model invocation, model fine-tuning, model deployment, model inference, and application. In some product forms, this computing device also supports model management, including but not limited to multi-type model management (supporting the management of discriminative, generative, and other types of models), model version control (supporting the control of different model versions), and model evaluation (evaluating model performance and effectiveness based on model evaluation tools). In other product forms, this computing device can also create applications based on models, providing API (Application Programming Interface) calling capabilities. Users can call models into created applications through the API interface, and application management tools are also provided to manage and monitor the applications.

[0251] Furthermore, the computing device may also include data management (supporting the creation and management of model tuning datasets), a training center (providing abundant training resources to help users learn and master AI (Artificial Intelligence) technology), and basic control capabilities (providing enterprise-level basic control capabilities to ensure the security and efficient operation of the system). Through the above functions, it provides a comprehensive and integrated device for AI development, training, deployment, and application.

[0252] Figure 7 A structural block diagram of an electronic device 700 provided according to one embodiment of this specification is shown.

[0253] A memory 710 and a processor 720 are connected via a bus 730;

[0254] The memory 710 is used to store computer programs / instructions, and the processor 720 is used to execute the computer programs / instructions, which, when executed by the processor 720, implement the steps of the method.

[0255] Specifically, the components of the electronic device 700 include, but are not limited to, a memory 710 and a processor 720. The processor 720 is connected to the memory 710 via a bus 730, and the database 750 is used to store data.

[0256] Electronic device 700 also includes access device 740, which enables electronic device 700 to communicate via one or more networks 760. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. Access device 740 may include one or more of any type of wired or wireless network interface (e.g., network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, Wi-MAX (Worldwide Interoperability for Microwave Access) interface, Ethernet interface, Universal Serial Bus (USB) interface, cellular network interface, Bluetooth interface, Near Field Communication (NFC) interface, and so on.

[0257] In one embodiment of this specification, the above-described components of the electronic device 700 and Figure 7 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 7The block diagram of the electronic device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0258] Electronic device 700 can be any type of stationary or mobile electronic device, including mobile computers or mobile electronic devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable electronic devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary electronic devices such as desktop computers or personal computers (PCs). Electronic device 700 can also be a mobile or stationary server.

[0259] The above is an illustrative scheme of an electronic device according to this embodiment. It should be noted that the technical solution of this electronic device and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the electronic device, please refer to the description of the technical solution of the data processing method described above.

[0260] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the above-described data processing method.

[0261] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the data processing method described above.

[0262] An embodiment of this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described data processing method.

[0263] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the data processing method described above.

[0264] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0265] The computer program / instructions include computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or system capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0266] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0267] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0268] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A data processing method, comprising: The target object is obtained from multiple videos from multiple perspectives. From the multiple videos from multiple perspectives, key video frames of each video are obtained. The multiple videos are time-synchronized, and the key video frames of each video correspond to the same time. Based on the object parameters of the target object in the key video frames of each video, construct a three-dimensional model of the target object at the key video frame; The 3D model of the target object at the key video frame and the position information of the key video frame and other video frames are sent to the client. The other video frames are the video frames in each video except for the key video frame. The 3D model and the position information are used to enable the client to obtain the 3D model of the target object at the other video frames, and to obtain the target video based on the 3D model of the target object at the key video frame and the 3D model of the target object at the other video frames.

2. The data processing method according to claim 1, wherein acquiring multiple videos of the target object from multiple perspectives includes: The system receives multiple videos of the target object sent by multiple image acquisition devices from different perspectives, wherein each video consists of multiple video frames.

3. The data processing method according to claim 1 or 2, wherein obtaining key video frames from multiple videos from the multiple perspectives comprises: An initial video is determined from multiple videos from the multiple perspectives, and at least one key video frame of the initial video is obtained; Based on the reference time corresponding to the at least one key video frame, the video frame corresponding to the reference time in each video is determined as the key video frame.

4. The data processing method according to claim 3, wherein obtaining at least one key video frame of the initial video comprises: The at least one key video frame is determined from the plurality of video frames based on the degree of change between the plurality of video frames of the initial video.

5. The data processing method according to claim 4, wherein determining the plurality of key video frames from the plurality of video frames based on the degree of change between the plurality of video frames of the initial video comprises: From the multiple video frames of the initial video, select and mark one video frame as the current key video frame, and then sequentially determine the other video frames from the multiple video frames as the current video frames; The degree of change between the current video frame and the current key video frame is determined sequentially, and the current video frame whose degree of change exceeds a preset threshold is updated and marked as the current key video frame. The multiple key video frames are obtained based on the multiple video frames that are marked as the current key video frames.

6. The data processing method according to claim 5, wherein selecting and marking a video frame as the current key video frame from the plurality of video frames of the initial video comprises: The first video frame among the multiple video frames of the initial video is determined as the current key video frame; The step of updating and marking the current video frame whose change exceeds a preset threshold as the current key video frame includes: If the degree of change exceeds the preset threshold, the current video frame is updated and marked as the current key video frame, and the steps of sequentially determining the degree of change between the current video frame and the current key video frame, and updating and marking the current video frame whose degree of change exceeds the preset threshold as the current key video frame are continued, until the degree of change between the current video frame and the current key video frame is determined.

7. The data processing method according to any one of claims 2, 4-6, wherein constructing a three-dimensional model of the target object at the key video frame based on the object parameters of the target object in the key video frame of each video comprises: Based on the device parameters corresponding to the multiple image acquisition devices and the object parameters of the target object in the key video frames of each video, the Gaussian distribution parameters corresponding to the key video frames are obtained. Based on the Gaussian distribution parameters corresponding to the key video frame, a three-dimensional Gaussian model of the target object at the key video frame is constructed.

8. The data processing method according to claim 7, wherein obtaining the Gaussian distribution parameters corresponding to the key video frames based on the device parameters corresponding to the plurality of image acquisition devices and the object parameters of the target object in the key video frames of each video includes: Using the multiple videos, obtain the device parameters corresponding to the multiple image acquisition devices; Based on the distortion coefficients in the device parameters corresponding to the multiple image acquisition devices, distortion removal processing is performed on the video corresponding to the image acquisition devices to obtain multiple processed videos. Based on the intrinsic and extrinsic parameters of the corresponding device parameters of each image acquisition device and the object parameters of the target object in the key video frames of the processed multiple videos, the Gaussian distribution parameters corresponding to the key video frames are obtained.

9. A data processing method, applied to a client, comprising: Receive a 3D model of the target object at a key video frame, and positional information between the key video frame and other video frames, wherein the key video frame is a key video frame of the target object in multiple videos from multiple perspectives, and the other video frames are video frames in the multiple videos other than the key video frame; The 3D model at the key video frame and the location information are input into the target parameter prediction network to obtain the 3D model of the target object at other video frames. The target video is obtained based on the 3D model of the target object at the key video frame and the 3D model at other video frames.

10. The data processing method according to claim 9, wherein inputting the 3D model at the key video frame and the location information into a target parameter prediction network to obtain the 3D model of the target object at the other video frames comprises: The Gaussian distribution parameters corresponding to the 3D model at the key video frame and the location information are input into the target parameter prediction network to obtain the Gaussian distribution parameters of the target object at other video frames. Based on the Gaussian distribution parameters of the target object in other video frames, a three-dimensional Gaussian model of the target object in other video frames is obtained.

11. The data processing method according to claim 9 or 10, wherein obtaining the target video based on the 3D model of the target object at the key video frame and the 3D model at the other video frames comprises: Based on the 3D model of the target object at the key video frame and the 3D model at other video frames, obtain the 3D model of the target object at each video frame; Based on the 3D model of the target object at each video frame and the time information, the volumetric video corresponding to the target object is obtained, wherein the time information is used to represent the timestamp of the video frame; The video of the specified volume is rendered to obtain the target video.

12. The data processing method according to claim 9 or 10, wherein the target parameter prediction network is trained through the following steps: Obtain multiple video samples of the object sample from multiple perspectives, and construct a three-dimensional model sample of the object sample at each video frame sample based on the object parameters of the object sample in the video frame sample of each video sample. From the video frame samples of each video sample, determine the key video frame samples and other video frame samples of each video sample; The Gaussian distribution parameters corresponding to the key video frame samples and the 3D model samples, as well as the position information of the key video frame samples and other video frame samples, are input into the initial parameter prediction network to obtain the predicted Gaussian distribution parameters of the other video frame samples. Based on the Gaussian distribution parameters corresponding to the other video frame samples and the 3D model samples, as well as the predicted Gaussian distribution parameters, the initial parameter prediction network is trained to obtain the target parameter prediction network.

13. A data processing system, comprising a server and a client; The server is used to acquire multiple videos of the target object from multiple perspectives, and to extract key video frames from each video from the multiple videos from multiple perspectives. The multiple videos are time-synchronized, and the key video frames of each video correspond to the same time. Based on the object parameters of the target object in the key video frames of each video, a three-dimensional model of the target object at the key video frame is constructed. The three-dimensional model of the target object at the key video frame and the position information between the key video frame and other video frames are sent to the client, wherein the other video frames are video frames in each video other than the key video frame. The client is configured to receive a 3D model of the target object at a key video frame, and positional information between the key video frame and other video frames; input the 3D model at the key video frame and the positional information into a target parameter prediction network to obtain a 3D model of the target object at the other video frames; and obtain a target video based on the 3D model of the target object at the key video frame and the 3D model at the other video frames.

14. A computing device, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 12.

15. An electronic device comprising: A memory and a processor, the memory and the processor being connected via a bus; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 12.

16. A computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 12.

17. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 12.

Images