Method and system for processing video elements in 3D scene

Abstract

The invention provides a method and a system for processing video elements in a 3D scene. The method comprises the following steps: acquiring a multi-view synchronous image sequence of a target reality scene; constructing 3D data of a target real scene by using a stereo matching technology, a deep learning algorithm and a multi-view synchronous image sequence; determining a dynamic object in the 3D data by using an optical flow estimation algorithm and an object recognition model; based on the dynamic object, determining spatial-temporal characteristics of the dynamic object in the 3D data; according to the method, 3D rendering and animation simulation are carried out based on the spatio-temporal characteristics and the 3D data, comprehensive and efficient technical support is provided for application of the digital twinborn technology in dynamic complex scenes, and the real-time performance, accuracy and practicability of a digital twinborn model are greatly improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to a method and system for processing video elements in a 3D scene. Background Art

[0002] Against the backdrop of the current convergence of digitalization and intelligent development, digital twin technology, as a core enabling technology, is gradually penetrating multiple fields, including industrial manufacturing, smart cities, and healthcare. Its core goal is to create a virtual replica that accurately corresponds to the physical world, enabling comprehensive monitoring, prediction, and optimization of entities, processes, and even systems. However, in 3D scenes, especially for complex and dynamically changing environments, effectively integrating and processing video elements poses a key technical challenge in building high-fidelity digital twin models.

[0003] Traditional 3D modeling and rendering methods often rely on static data and find it difficult to capture and reflect dynamic changes in the real world in real time, which limits the potential of digital twin technology in dynamic scene applications.

[0004] Existing technical solutions mostly focus on a single dimension, such as isolated 3D modeling or moving object detection. They lack a systematic framework to comprehensively process the spatiotemporal information in video data and integrate it into the real-time simulation and analysis of digital twins. Furthermore, ensuring high accuracy while achieving efficient algorithm execution and reducing computing resource consumption is also a pressing issue. Summary of the Invention

[0005] The embodiments of the present application provide a method for processing video elements in a 3D scene to solve the problems existing in the above-mentioned background technology.

[0006] In a first aspect, an embodiment of the present application provides a method for processing video elements in a 3D scene, comprising: obtaining a multi-perspective synchronized image sequence of a target real scene; constructing 3D data of the target real scene using stereo matching technology, a deep learning algorithm and the multi-perspective synchronized image sequence, wherein the 3D data includes: a 3D point cloud model or a voxel grid; determining dynamic objects in the 3D data using an optical flow estimation algorithm and an object recognition model; determining the spatiotemporal characteristics of the dynamic objects in the 3D data based on the dynamic objects, wherein the spatiotemporal characteristics include at least: motion trajectory, motion speed, motion acceleration and interaction between dynamic objects; performing 3D rendering and animation simulation based on the spatiotemporal characteristics and the 3D data.

[0007] Furthermore, the 3D data of the target real scene is constructed using stereo matching technology, a deep learning algorithm and the multi-perspective synchronized image sequence, including: performing a feature detection algorithm on the corresponding images in the multi-perspective synchronized image sequence to obtain feature points, and matching the feature points to obtain matching feature points; using the stereo matching technology, performing disparity calculation on the matching feature points, and constructing a disparity map based on the disparity calculation results; using a deep neural network, correcting the disparity map to obtain a target disparity map; and constructing the 3D data of the target real scene based on the target disparity map.

[0008] Furthermore, the loss function of the deep neural network is: ;in, is the depth error term, is the smoothness constraint, is the structural similarity term, is the feature matching item, 、 、 and is the weight coefficient.

[0009] Furthermore, based on the target disparity map, 3D data of the target real scene is constructed, including: based on the parameters of the camera, converting the disparity of each pixel in the target disparity map into a depth value in the actual space to obtain a depth map, wherein the camera is a camera that shoots the multi-perspective synchronized image sequence, and the parameters include intrinsic parameters and extrinsic parameters; based on the depth value of each pixel in the depth map and the focal length and optical center coordinates of the camera, determining the coordinates of each pixel in the depth map in the three-dimensional space; based on the coordinates of each pixel in the depth map in the three-dimensional space, constructing the 3D data of the target real scene.

[0010] Furthermore, based on the camera parameters, the disparity of each pixel in the target disparity map is converted into a depth value in the real space to obtain a depth map, including: using the intrinsic parameters to convert the coordinates of each pixel in the target disparity map into normalized image coordinates; on the normalized plane, using a preset calculation formula to determine the depth map; the preset calculation formula is: ;in, is the depth value, is parallax, is the focal length, is the distance between the two camera centers corresponding to the target disparity map.

[0011] Furthermore, the optical flow estimation algorithm and the object recognition model are used to determine the dynamic objects in the 3D data, including: using the optical flow estimation algorithm to determine the dynamic pixels in the multi-view synchronized image sequence; inputting the 3D data into the object recognition model to obtain a recognition result; and determining the dynamic objects based on the recognition result and the dynamic pixels.

[0012] Furthermore, the optical flow estimation algorithm is used to determine the dynamic pixels in the multi-perspective synchronized image sequence, including: calculating the pixel intensity difference between two consecutive frames of each perspective image sequence in the multi-perspective synchronized image sequence; and determining the dynamic pixels based on the pixel intensity difference and the optical flow estimation algorithm.

[0013] In a second aspect, an embodiment of the present application provides a system for processing video elements in a 3D scene, comprising: an acquisition unit for acquiring a multi-perspective synchronized image sequence of a target real scene; a construction unit for constructing 3D data of the target real scene using stereo matching technology, a deep learning algorithm and the multi-perspective synchronized image sequence, wherein the 3D data includes: a 3D point cloud model or a voxel grid; an identification unit for determining dynamic objects in the 3D data using an optical flow estimation algorithm and an object recognition model; a determination unit for determining the spatiotemporal characteristics of dynamic objects in the 3D data based on the dynamic objects, wherein the spatiotemporal characteristics include at least: motion trajectory, motion speed, motion acceleration and interaction between dynamic objects; a processing unit for performing 3D rendering and animation simulation based on the spatiotemporal characteristics and the 3D data.

[0014] In a third aspect, an embodiment of the present application provides a computing device comprising a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement the method described in the first aspect above.

[0015] In a fourth aspect, an embodiment of the present application provides a computer storage medium storing a computer program, which, when executed by a computer, implements the method described in the first aspect above.

[0016] In an embodiment of the present invention, by integrating synchronous multi-perspective image acquisition, stereo matching and deep learning-driven 3D reconstruction, precise recognition of dynamic objects and spatiotemporal feature extraction, and 3D dynamic rendering and simulation based on these features, comprehensive and efficient technical support is provided for the application of digital twin technology in dynamic and complex scenes, and the real-time, accuracy and practicality of the digital twin model are greatly improved.

[0017] These and other aspects of the present application will become more readily apparent from the description of the following embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0019] Figure 1 A flowchart of a method for processing video elements in a 3D scene provided in an embodiment of the present application; Figure 2 A schematic diagram of a system for processing video elements in a 3D scene provided in an embodiment of the present application; Figure 3 A schematic diagram of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0021] Example 1: According to an embodiment of the present invention, an embodiment of a method for processing video elements in a 3D scene is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0022] Figure 1 FIG. 1 is a flow chart of a method for processing video elements in a 3D scene according to an embodiment of the present invention. Figure 1 As shown, the method includes the following steps: Step S102, acquiring a multi-perspective synchronized image sequence of a target real scene; Step S104, constructing 3D data of the target real scene using stereo matching technology, a deep learning algorithm, and the multi-view synchronized image sequence, wherein the 3D data includes: a 3D point cloud model or a voxel grid; Step S106, using an optical flow estimation algorithm and an object recognition model to determine dynamic objects in the 3D data; Step S108, determining the spatiotemporal characteristics of the dynamic object in the 3D data based on the dynamic object, wherein the spatiotemporal characteristics at least include: motion trajectory, motion speed, motion acceleration, and interaction between the dynamic object; Step S110 : performing 3D rendering and animation simulation based on the spatiotemporal features and the 3D data.

[0023] The following uses an automated production line in a smart factory as an example to illustrate the above steps.

[0024] It should be noted that the production line includes multiple automatic robotic arms, conveyor belts, and material boxes that move between different workstations.

[0025] Multiple high-definition cameras installed around the production line capture real-time images of the production line according to preset viewing angles and synchronization mechanisms. These cameras ensure coverage of all key areas of the production line, capturing multiple frames per second to form a series of synchronized image sequences.

[0026] Stereo matching technology, combined with deep learning algorithms (such as convolutional neural networks (CNNs)), processes these multi-view image sequences to generate a 3D point cloud model of the production line. This model accurately maps the production line layout, including the location and shape of machinery and equipment. For further refinement, the point cloud can be converted into a voxel grid for improved rendering efficiency and enhanced detail.

[0027] Use the optical flow estimation algorithm to track the changes in pixels between consecutive frames, combined with a pre-trained object recognition model (such as YOLOv5), to identify and isolate dynamic objects, such as the movement of robot arms and the movement of material boxes on conveyor belts.

[0028] For each identified dynamic object, its trajectory, average velocity, and acceleration are calculated. Furthermore, by analyzing the relative positional changes between dynamic objects, interactions between them can be identified, such as when a robot arm picks up and places a material bin. This information helps understand production line efficiency and potential bottlenecks.

[0029] Based on these spatiotemporal characteristics and the 3D model, a digital twin of the production line is created using advanced rendering engines such as Unity or Unreal Engine. Within this virtual environment, the precise movements of each robotic arm, the movement trajectory of the material bins, and their interactions are dynamically simulated. These dynamic simulations are displayed on a real-time monitoring interface. The system also predicts the production line's status over time, assisting management in optimizing decisions.

[0030] This approach enables factory managers to intuitively monitor the real-time status of the entire production line, identifying inefficiencies or potential failure points without requiring on-site inspections. By analyzing the spatiotemporal characteristics of dynamic objects, production bottlenecks can be quickly identified, workflows can be optimized, and maintenance activities can be planned in advance, thereby reducing downtime and improving production efficiency. Furthermore, this system provides powerful technical support for remote monitoring and intelligent decision-making, meeting the requirements of modern smart factories for efficient management and predictive maintenance.

[0031] In an embodiment of the present invention, the 3D data of the target real scene is constructed using stereo matching technology, a deep learning algorithm, and the multi-view synchronized image sequence, including: Performing a feature detection algorithm on corresponding images in the multi-view synchronized image sequence to obtain feature points, and matching the feature points to obtain matching feature points; Utilizing the stereo matching technology, performing disparity calculation on the matching feature points, and constructing a disparity map based on the disparity calculation results; Using a deep neural network, the disparity map is corrected to obtain a target disparity map; Based on the target disparity map, 3D data of the target real scene is constructed.

[0032] In this embodiment of the present invention, high-definition cameras are first selected and distributed around the production line, ensuring that they simultaneously capture key areas of the line. Feature detection algorithms such as SIFT (Scale-Invariant Feature Transform) or ORB (Oriented FAST and Rotated BRIEF) are used to extract feature points from the image sequences captured by each camera. Subsequently, a feature matching algorithm (such as BFMatcher or FLANN matcher) is used to find corresponding feature point pairs between images from different viewpoints. This step is crucial for subsequent disparity calculations, as it helps establish the spatial relationship between multi-view images.

[0033] After obtaining the matched feature points, stereo matching techniques, such as block matching or semi-global matching (SGM), are applied to calculate the disparity between these matched point pairs. Disparity reflects the positional differences between the same physical point in different images and is directly related to the depth information of that point. All disparity values are integrated into a disparity map, in which the color or grayscale value of each pixel represents the depth information of that pixel in three-dimensional space.

[0034] Because basic disparity calculation methods can be affected by noise, occlusion, and discontinuities, errors can occur in the disparity map. To address this, a pre-trained deep neural network, such as U-Net or DeepPruner, is used to post-process the initially constructed disparity map. By learning from a large dataset with accurate depth labels, this network can identify and correct errors in the disparity map, outputting a more accurate target disparity map. This step significantly improves the accuracy and reliability of 3D reconstruction.

[0035] Preferably, in an embodiment of the present invention, the loss function of the deep neural network is: ; in, is the depth error term, is the smoothness constraint, is the structural similarity term, is the feature matching item, 、 、 and is the weight coefficient.

[0036] Specifically, , For the The true depth value of each pixel, For the The depth value of pixels obtained by parallax conversion, is the total number of pixels, is a weight function that decreases with increasing depth, Controls the decay speed.

[0037] , and They are the pixel steps in the horizontal and vertical directions, which can be adjusted according to actual conditions.

[0038] , is an image reconstructed based on the estimated depth, It is a real image.

[0039] , Indicates the layer feature extractor, , , are the number of channels, height, and width of the feature map of this layer, respectively.

[0040] Finally, based on the resulting target disparity map, inverse depth mapping or other 3D reconstruction algorithms are used to convert the disparity information into actual 3D spatial coordinates, constructing a 3D point cloud model of the production line. This model not only captures the precise geometry of the production line equipment but also accurately reflects the relative positions and sizes of dynamic objects on the line, such as robots and material bins. Furthermore, point cloud-to-mesh conversion techniques can be used to generate even more detailed voxel mesh models, providing high-fidelity 3D scenes for digital twin applications.

[0041] Through the above steps, smart factories can obtain high-precision 3D models of production lines, which is crucial for achieving refined management and optimization. For example, this 3D data can be used for virtual simulation of production lines, predicting and optimizing production processes, avoiding physical collisions, and improving production efficiency. Furthermore, combined with real-time monitoring data, fault points can be quickly identified, enabling preventive maintenance, reducing downtime risks, and improving overall operational efficiency. For remote monitoring and training, realistic 3D models also provide an intuitive learning and decision-making support environment, facilitating knowledge transfer and skills training.

[0042] In an embodiment of the present invention, in the scenario of an automated production line in a smart factory, the process from constructing a target disparity map to constructing 3D data is as follows: Each camera's intrinsic parameters (such as focal length f, optical center coordinates (cx, cy)) and extrinsic parameters (such as rotation matrix R and translation vector T) are known. These parameters are usually obtained during the camera calibration process. Using this information, the disparity value d of each pixel in the target disparity map is converted to the actual depth value Z. The conversion formula is based on basic trigonometric principles and can be expressed as: ,in, is the depth value, is parallax, is the focal length, is the distance between the two camera centers corresponding to the target disparity map.

[0043] After obtaining the depth map, the (u, v) coordinates of each pixel (i.e., the horizontal and vertical coordinates in the image coordinate system) and the focal length and optical center coordinates of the camera can be combined to calculate the (X, Y, Z) coordinates of each pixel in three-dimensional space. The three-dimensional position of each pixel in the real world is determined, forming a dense three-dimensional point cloud.

[0044] By combining all the resulting 3D coordinate points, a 3D point cloud model of the smart factory's automated production line is constructed. This point cloud model provides 3D geometric information about the production line, including its layout, equipment locations, and conveyor belt paths. To improve the model's practicality and readability, the point cloud can be further processed through methods such as denoising, clustering, and surface reconstruction to create a smoother and more structured 3D model. This can be achieved using open-source libraries such as PCL (Point Cloud Library).

[0045] In a smart factory environment, building 3D data of production lines has significant practical value: Optimize layout and planning: 3D models help simulate different production line layout options, evaluate their efficiency and feasibility in advance, and reduce the cost and time of physical adjustments.

[0046] Robot navigation and obstacle avoidance: Provides automated robots with accurate environmental maps, enabling them to effectively plan paths, avoid collisions, and improve operational safety and efficiency.

[0047] Remote monitoring and maintenance: Remote operators can monitor production line status through 3D views, quickly locate problems, and provide remote guidance or repairs, reducing the need for on-site personnel.

[0048] Quality control and analysis: Combined with sensor data, 3D models help analyze anomalies in the production process, optimize process flow, and improve product quality.

[0049] In an embodiment of the present invention, determining dynamic objects in the 3D data using an optical flow estimation algorithm and an object recognition model includes: Determining dynamic pixels in the multi-view synchronized image sequence using the optical flow estimation algorithm; Inputting the 3D data into the object recognition model to obtain a recognition result; The dynamic object is determined based on the recognition result and the dynamic pixels.

[0050] In the context of smart factory automation production lines, the specific implementation steps for using optical flow estimation algorithms and object recognition models to identify dynamic objects in 3D data are as follows: First, optical flow estimation algorithms, such as the Lucas-Kanade method or modern algorithms like DeepFlow, are applied to multi-view synchronized image sequences from the automated production line in a smart factory. These algorithms estimate the motion vector (i.e., optical flow) for each pixel in the image sequence by comparing the positional changes of pixels between consecutive frames. In a production line environment, static backgrounds (such as fixed equipment and conveyor belts) exhibit low optical flow values, while dynamic objects (such as moving parts and products) produce noticeable optical flow motion. Therefore, a threshold can be set to filter out regions with high optical flow changes, representing dynamic pixels.

[0051] The identified dynamic areas are mapped to the 3D data, and the 3D point clouds of these areas are selected as input and fed into a pre-trained object recognition model. This model may be based on deep learning architectures such as YOLO and Mask R-CNN, which can identify and classify different objects. In smart factory scenarios, the model needs to be customized for common objects on the production line (such as parts to be processed, tools, and finished products) to ensure high recognition accuracy.

[0052] Combining the object recognition results with the dynamic pixels in the step can accurately locate which 3D point cloud areas belong to dynamic objects. For example, if the recognition model specifies that a certain area contains a moving part, the corresponding dynamic pixel area can be confirmed as a dynamic object. This process may also include post-processing of the recognition results, such as using non-maximum suppression (NMS) to remove overlapping detection boxes to ensure that each dynamic object can be uniquely identified and tracked.

[0053] Example 2: An embodiment of the present invention also provides a system for processing video elements in a 3D scene. The system for processing video elements in a 3D scene is used to execute the method for processing video elements in a 3D scene provided in the above content of the embodiment of the present invention. The following is a detailed introduction to the system for processing video elements in a 3D scene provided by an embodiment of the present invention.

[0054] like Figure 2 As shown, Figure 2 Schematic diagram of a system for processing video elements in the above 3D scene, the system comprising: An acquisition unit 10 is configured to acquire a multi-perspective synchronized image sequence of a target real scene; A construction unit 20 is configured to construct 3D data of the target real scene using stereo matching technology, a deep learning algorithm, and the multi-view synchronized image sequence, wherein the 3D data includes: a 3D point cloud model or a voxel grid; an identification unit 30 for determining dynamic objects in the 3D data using an optical flow estimation algorithm and an object recognition model; A determining unit 40 is configured to determine spatiotemporal features of the dynamic object in the 3D data based on the dynamic object, wherein the spatiotemporal features include at least: motion trajectory, motion speed, motion acceleration, and interaction between the dynamic objects; The processing unit 50 is configured to perform 3D rendering and animation simulation based on the spatiotemporal features and the 3D data.

[0055] In an embodiment of the present invention, by integrating synchronous multi-perspective image acquisition, stereo matching and deep learning-driven 3D reconstruction, precise recognition of dynamic objects and spatiotemporal feature extraction, and 3D dynamic rendering and simulation based on these features, comprehensive and efficient technical support is provided for the application of digital twin technology in dynamic and complex scenes, and the real-time, accuracy and practicality of the digital twin model are greatly improved.

[0056] Example 3: The embodiment of the present invention further provides a computing device for executing the program of the method described in the above embodiment 1, such as Figure 3 As shown, the computing device may include a storage component 41 and a processing component 42; The storage component 41 stores one or more computer instructions, wherein the one or more computer instructions are called and executed by the processing component 42 .

[0057] The processing component 42 may include one or more processors to execute computer instructions to complete all or part of the steps of the method of Example 1. Of course, the processing component may also be implemented as one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above method.

[0058] The storage component 41 is configured to store various types of data to support operations at the terminal. The storage component can be implemented by any type of volatile or non-volatile memory device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0059] The display component 43 may be an electroluminescent (EL) element, a liquid crystal display or a micro display having a similar structure, or a retinal direct display or a similar laser scanning display.

[0060] Of course, a computing device may also include other components, such as input / output interfaces, communication components, etc.

[0061] The input / output interface provides an interface between the processing component and the peripheral interface module, which can be an output device, an input device, etc.

[0062] The communication component is configured to facilitate, among other things, wired or wireless communications between the computing device and other devices.

[0063] Among them, the computing device can be a physical device or an elastic computing host provided by a cloud computing platform, etc. In this case, the computing device can refer to a cloud server, and the above-mentioned processing components, storage components, etc. can be basic server resources rented or purchased from the cloud computing platform.

[0064] Example 4: The present application also provides a computer storage medium storing a computer program, wherein the computer program can achieve the above-mentioned Figure 1 The method of the embodiment shown.

[0065] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0066] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0067] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0068] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for processing video elements in a 3D scene, characterized in that: include: Acquire multi-perspective synchronized image sequences of the target real scene; Constructing 3D data of the target real scene using stereo matching technology, a deep learning algorithm, and the multi-view synchronized image sequence, wherein the 3D data includes: a 3D point cloud model or a voxel grid; Determining dynamic objects in the 3D data using an optical flow estimation algorithm and an object recognition model; Based on the dynamic object, determining spatiotemporal features of the dynamic object in the 3D data, wherein the spatiotemporal features include at least: motion trajectory, motion speed, motion acceleration, and interaction between the dynamic object; 3D rendering and animation simulation are performed based on the spatiotemporal features and the 3D data.

2. The method according to claim 1, characterized in that Utilizing stereo matching technology, a deep learning algorithm, and the multi-view synchronized image sequence to construct 3D data of the target real scene, including: Performing a feature detection algorithm on corresponding images in the multi-view synchronized image sequence to obtain feature points, and matching the feature points to obtain matching feature points; Utilizing the stereo matching technology, performing disparity calculation on the matching feature points, and constructing a disparity map based on the disparity calculation results; Using a deep neural network, the disparity map is corrected to obtain a target disparity map; Based on the target disparity map, 3D data of the target real scene is constructed.

3. The method according to claim 2, characterized in that The loss function of the deep neural network is: ； in, is the depth error term, is the smoothness constraint, is the structural similarity term, is the feature matching item, 、 、 and is the weight coefficient.

4. The method according to claim 2, characterized in that Constructing 3D data of the target real scene based on the target disparity map, including: Based on camera parameters, converting the disparity of each pixel in the target disparity map into a depth value in real space to obtain a depth map, wherein the camera is a camera that captures the multi-view synchronized image sequence, and the parameters include intrinsic parameters and extrinsic parameters; Determining the coordinates of each pixel in the depth map in three-dimensional space based on the depth value of each pixel in the depth map and the focal length and optical center coordinates of the camera; Based on the coordinates of each pixel in the depth map in three-dimensional space, 3D data of the target real scene is constructed.

5. The method according to claim 4, characterized in that Based on the camera parameters, the disparity of each pixel in the target disparity map is converted into a depth value in the real space to obtain a depth map, including: Converting the coordinates of each pixel in the target disparity map into normalized image coordinates using the intrinsic reference; On the normalized plane, determining the depth map using a preset calculation formula; The preset calculation formula is: ； in, is the depth value, is parallax, is the focal length, is the distance between the two camera centers corresponding to the target disparity map.

6. The method according to claim 1, characterized in that Determining dynamic objects in the 3D data using an optical flow estimation algorithm and an object recognition model includes: Determining dynamic pixels in the multi-view synchronized image sequence using the optical flow estimation algorithm; Inputting the 3D data into the object recognition model to obtain a recognition result, wherein the recognition result is used to characterize the object contained in the target real scene; The area in the dynamic pixels corresponding to the recognition result is determined as the dynamic object.

7. The method according to claim 6, characterized in that Determining dynamic pixels in the multi-view synchronized image sequence using the optical flow estimation algorithm includes: Calculating a pixel intensity difference between two consecutive frames of each perspective image sequence in the multi-perspective synchronized image sequence; Determining a motion vector for each pixel based on the pixel intensity difference and an optical flow estimation algorithm; Pixels with motion vectors greater than a preset threshold are determined as dynamic pixels.

8. A system for processing video elements in a 3D scene, characterized in that: include: An acquisition unit, configured to acquire a multi-perspective synchronized image sequence of a target real scene; A construction unit, configured to construct 3D data of the target real scene using stereo matching technology, a deep learning algorithm, and the multi-view synchronized image sequence, wherein the 3D data includes: a 3D point cloud model or a voxel grid; an identification unit, configured to identify dynamic objects in the 3D data using an optical flow estimation algorithm and an object recognition model; a determining unit, configured to determine, based on the dynamic object, spatiotemporal features of the dynamic object in the 3D data, wherein the spatiotemporal features include at least: motion trajectory, motion speed, motion acceleration, and interaction between the dynamic objects; A processing unit is used to perform 3D rendering and animation simulation based on the spatiotemporal characteristics and the 3D data.

9. A computing device, characterized in that It comprises a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement the method according to any one of claims 1 to 7.

10. A computer storage medium, characterized in that A computer program is stored, and when the computer program is executed by a computer, the method according to any one of claims 1 to 7 is implemented.

Images