Video-driven vector data 3Dization and real-time rendering method and system based on digital twin scene

Abstract

The application provides a video-driven vector data three-dimensionalization and real-time rendering method and system based on a digital twin scene, and relates to the technical field of three-dimensional reconstruction. The method comprises the following steps: firstly, collecting initial video streams and point cloud data of a mobile carrier and performing semantic segmentation to obtain initial vector data of a dynamic target; then, generating control instructions based on the vector data and real-time poses of the carrier to adjust the collection parameters to obtain optimized video streams and point cloud data of the dynamic target; further, generating fusion feature data in combination with a time sequence fusion network; finally, driving a neural radiation field to perform three-dimensional reconstruction and rendering to generate a real-time rendering result. The application improves the accuracy and real-time performance of three-dimensional reconstruction of a dynamic target in a complex scene.

Description

Technical Field

[0001] This application relates to the field of 3D reconstruction technology, and in particular to a method and system for video-driven vector data 3D generation and real-time rendering based on digital twin scenes. Background Technology

[0002] Video-driven 3D reconstruction and real-time rendering technology is key to realizing dynamic digital twin scenes. This method generates or updates 3D models of scenes by analyzing visual data such as video streams, and has broad application prospects in fields such as urban management and intelligent transportation.

[0003] Existing methods are typically deployed on mobile platforms and use cameras and sensors such as LiDAR to collect environmental data synchronously. The acquired point clouds and images are then processed separately. For example, some solutions first perform semantic segmentation on the point cloud data to identify different object categories, then fuse this semantic information with video frames, and finally use 3D reconstruction techniques such as neural radiation fields to generate the scene rendering results.

[0004] However, when dealing with large-scale dynamic scenes, these methods rely on the quality of the initial data obtained under a fixed acquisition strategy for the perception and reconstruction of moving targets. When the target moves quickly in the scene or has complex interactions with the background, the coverage and specificity of the initial data are insufficient, which may lead to the loss of geometric details or blurring of the appearance of the dynamic target in the final 3D model, affecting the accuracy of the rendering results. Therefore, the existing technology has the technical problem of limited accuracy and real-time performance in the 3D reconstruction of dynamic targets. Summary of the Invention

[0005] This application provides a method and system for video-driven vector data 3D reconstruction and real-time rendering based on digital twin scenes, in order to solve the problems of low precision and poor real-time performance of dynamic target 3D reconstruction in complex scenes in the prior art.

[0006] To address the aforementioned technical problems, in a first aspect, this application provides a method for video-driven vector data 3D generation and real-time rendering based on a digital twin scene, comprising:

[0007] In a digital twin scenario, the initial video stream and initial point cloud data of the mobile carrier are collected;

[0008] The initial point cloud data is semantically segmented to generate initial semantic information, and initial vector data of the dynamic targets is generated based on the category labels of the dynamic targets in the initial semantic information.

[0009] Based on the initial vector data and combined with the real-time pose information of the mobile carrier, control commands are generated.

[0010] According to the control command, the acquisition parameters are adjusted to obtain the target video stream and target point cloud data for the dynamic target;

[0011] Based on the target vector data corresponding to the target video stream and the target point cloud data, and combined with a temporal fusion network, fused feature data is generated.

[0012] Based on the fused feature data, the neural radiation field is driven to perform three-dimensional reconstruction and rendering, generating real-time rendering results.

[0013] Optionally, the step of driving the neural radiation field to perform three-dimensional reconstruction and rendering based on the fused feature data, and generating real-time rendering results, includes:

[0014] The fused feature data and the target vector data are input into the neural radiation field, and a dynamic target region is determined in the neural radiation field based on the target vector data.

[0015] Three-dimensional reconstruction was performed on sampling points located within and outside the dynamic target area to obtain the reconstruction results;

[0016] Based on the reconstruction results, dynamic image regions and static image regions are rendered respectively;

[0017] The dynamic image region and the static image region are combined at the pixel level to generate a real-time rendering result.

[0018] Optionally, the step of performing three-dimensional reconstruction on sampling points located within and outside the dynamic target area to obtain reconstruction results includes:

[0019] Based on the target vector data, determine the location attributes of each sampling point;

[0020] For sampling points whose location attribute is a dynamic target area, a first density value is obtained by querying a preset first density table, and a first color value is obtained by querying a preset first color table.

[0021] For sampling points whose location attribute is outside the dynamic target area, a second density value is obtained by querying a preset second density table, and a second color value is obtained by querying a preset second color table.

[0022] Along a preset observation ray, the first density value and the first color value are calculated to generate a first reconstruction result;

[0023] The second density value and the second color value are calculated along the observed light ray to generate a second reconstruction result.

[0024] Optionally, the step of generating fused feature data based on the target vector data corresponding to the target video stream and the target point cloud data, combined with a temporal fusion network, includes:

[0025] The target point cloud data is processed to generate target vector data;

[0026] Select keyframes that are time-aligned with the target vector data from the target video stream;

[0027] The target vector data and the keyframes are input into a temporal fusion network. The extraction module of the temporal fusion network extracts image features from the video keyframe sequence and spatial features from the target vector data.

[0028] The adjustment module of the temporal fusion network adjusts the weights of the image features in the image space based on the spatial features.

[0029] The output module of the temporal fusion network concatenates and transforms the weighted image features and spatial features along the channel dimension to output fused feature data.

[0030] Optionally, the step of generating control commands based on the initial vector data and combined with the real-time pose information of the mobile carrier includes:

[0031] The real-time pose information of the mobile carrier is obtained, and the real-time pose information includes the position and orientation information of the mobile carrier at the current moment;

[0032] Based on the initial vector data, determine the center position of the dynamic target in three-dimensional space;

[0033] Based on the position and orientation information of the mobile carrier, the observation direction of the sensor group installed on the mobile carrier is determined;

[0034] Calculate the angular difference and distance difference between the center position and the observation direction;

[0035] Based on the angle difference and the distance difference, and combined with the preset servo control logic, a servo compensation command is generated.

[0036] A path planning instruction pointing to the center location is generated based on a preset scene scanning path.

[0037] The servo compensation command and the path planning command are combined to form a control command.

[0038] Optionally, the step of performing semantic segmentation on the initial point cloud data to generate initial semantic information, and generating initial vector data of the dynamic targets based on the category labels of the dynamic targets in the initial semantic information, includes:

[0039] The initial point cloud data is processed to obtain the category probability of each point;

[0040] Based on the category probabilities, each point is assigned to a corresponding category and labeled with a category tag to form initial semantic information;

[0041] Filter out all points whose category labels belong to the preset dynamic target category from the initial semantic information;

[0042] Cluster the selected points to identify different dynamic targets;

[0043] For each dynamic target, the corresponding three-dimensional spatial boundary is determined, and initial vector data is generated based on the three-dimensional spatial boundary.

[0044] Optionally, adjusting the acquisition parameters according to the control command to obtain the target video stream and target point cloud data for the dynamic target includes:

[0045] The servo compensation command in the control command is sent to the sensor platform to drive the sensor platform to rotate so that the sensor observation direction is aligned with the center position.

[0046] After the sensor platform rotates into position, the laser radar is controlled to scan and the camera is controlled to acquire images according to the path planning instructions in the control commands.

[0047] While the sensor platform remains aligned, it continuously performs data acquisition operations for a preset duration to obtain target video streams and target point cloud data.

[0048] Secondly, this application provides a video-driven vector data 3D generation and real-time rendering system based on a digital twin scene, comprising:

[0049] The acquisition module is used to acquire the initial video stream and initial point cloud data of the mobile carrier in a digital twin scenario.

[0050] The segmentation module is used to perform semantic segmentation on the initial point cloud data, generate initial semantic information, and generate initial vector data of the dynamic targets based on the category labels of the dynamic targets in the initial semantic information.

[0051] The first generation module is used to generate control commands based on the initial vector data and the real-time pose information of the mobile carrier.

[0052] The adjustment module is used to adjust the acquisition parameters according to the control command in order to obtain the target video stream and target point cloud data for the dynamic target;

[0053] The second generation module is used to generate fused feature data based on the target vector data corresponding to the target video stream and the target point cloud data, combined with a temporal fusion network.

[0054] The driving module is used to drive the neural radiation field to perform three-dimensional reconstruction and rendering based on the fused feature data, and generate real-time rendering results.

[0055] Thirdly, this application provides an electronic device, comprising:

[0056] Memory, used to store computer programs;

[0057] A processor is configured to execute the computer program to implement the steps of the video-driven vector data 3D generation and real-time rendering method based on a digital twin scene as described in the first aspect above.

[0058] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps of the video-driven vector data 3D generation and real-time rendering method based on a digital twin scene as described in the first aspect above.

[0059] This application provides a method for video-driven vector data 3D generation and real-time rendering based on digital twin scenes, which has the following advantages:

[0060] First, acquiring initial video streams and point cloud data provides the basic scene perception input for subsequent processing. Then, semantic segmentation of the initial point cloud generates vector data of dynamic targets, thereby achieving structured recognition and localization of key moving objects in the scene. Next, control commands are generated based on the vector data and the carrier pose, guiding the sensor system to actively adjust. After adjusting the acquisition parameters according to the commands, target video streams and point cloud data optimized for dynamic targets can be obtained, improving the quality of the data source used for reconstruction. Subsequently, a temporal fusion network is used to process the target video streams and vector data to generate fusion features, enhancing the temporal and spatial correlation between different modalities. Finally, the fusion features drive the neural radiation field for reconstruction and rendering, achieving real-time and refined 3D presentation of the dynamic scene.

[0061] Furthermore, this method distinguishes between dynamic target regions and static background regions and adopts differentiated reconstruction strategies, enabling limited computing resources to prioritize the geometric details and appearance quality of dynamic targets while maintaining the overall scene rendering efficiency. Finally, by compositing the separately rendered image parts, the method effectively improves the visual detail and realism of dynamic objects in the final rendering result while ensuring scene integrity.

[0062] These or other aspects of this application will become more apparent in the following description of the embodiments. Attached Figure Description

[0063] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0064] Figure 1 A flowchart illustrating a method for video-driven vector data 3D generation and real-time rendering based on a digital twin scene, provided in this application embodiment;

[0065] Figure 2 A schematic diagram illustrating a specific implementation of a video-driven vector data 3D generation and real-time rendering method based on a digital twin scene, provided in this application embodiment;

[0066] Figure 3 This is a schematic diagram of the structure of a video-driven vector data 3D generation and real-time rendering system based on a digital twin scene, provided in an embodiment of this application. Detailed Implementation

[0067] In existing technologies, mobile platform-based 3D reconstruction methods typically acquire scene data based on a pre-defined fixed acquisition strategy and then perform subsequent processing. Therefore, when faced with complex dynamic scenes containing fast-moving targets, this static data acquisition method has the following inherent limitations: the initial data captured by the sensors is not optimized for the rapidly changing key dynamic targets in the scene; as a result, the core input data used for 3D reconstruction is insufficient in terms of coverage, density, and relevance, which directly restricts the geometric refinement and realistic appearance of the dynamic target areas in the final rendering result, thereby limiting the real-time expressiveness and practicality of the overall 3D model.

[0068] To overcome this deficiency, this invention proposes a video-driven vector data 3D reconstruction and real-time rendering method based on digital twin scenarios. The core of this approach lies in first rapidly identifying and structurally representing dynamic targets from initial sensing data, then generating dynamic control commands in real-time based on this representation information, thereby actively guiding the sensor system to adjust its acquisition posture and parameters. This closed-loop feedback process enables the proactive acquisition of optimized enhanced data for dynamic targets. Subsequently, temporal fusion technology is used to deeply correlate these enhanced multimodal data with the target's structured information, finally driving the 3D reconstruction model for rendering. This method effectively solves the problem of limited accuracy and real-time performance in dynamic target reconstruction caused by insufficient targeting of initial data by dynamically optimizing the data source quality of the 3D reconstruction process through real-time feedback of sensing results to the data acquisition front end.

[0069] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0070] The core of this application is to provide a method for video-driven vector data 3D generation and real-time rendering based on digital twin scenes. A flowchart of one specific implementation is shown below. Figure 1 As shown, the method includes:

[0071] Step 101: In the digital twin scenario, collect the initial video stream and initial point cloud data of the mobile carrier.

[0072] In step 101, the digital twin scene refers to the dynamic digital mapping of the real physical world in virtual space for simulation, analysis and control; the mobile vehicle refers to a platform that moves in the real scene, such as an inspection vehicle or a drone; the initial video stream is a sequence of images continuously captured by a camera on the mobile vehicle; the initial point cloud data is a set of a large number of three-dimensional spatial points on the surface of the surrounding environment measured by the lidar on the mobile vehicle by emitting laser beams and receiving reflected signals, which reflects the geometry of the scene.

[0073] In this embodiment of the application, the method is executed on a mobile carrier equipped with a camera and a lidar. The camera's image recording and the lidar's scanning are started simultaneously through a synchronous triggering mechanism, so that the initial video stream and initial point cloud data corresponding to the same moment and the same field of view are collected in real time and in parallel during the movement of the carrier.

[0074] Step 102: Perform semantic segmentation on the initial point cloud data to generate initial semantic information, and generate initial vector data of the dynamic targets based on the category labels of the dynamic targets in the initial semantic information.

[0075] Among them, the initial semantic information is an enhanced form of the initial point cloud data. It adds the category label information assigned to each point while retaining the original three-dimensional coordinates of each point. The category label is a symbol used to uniquely identify the category to which a point belongs, such as "vehicle" or "pedestrian". The initial vector data is a mathematical description of the boundary of the dynamic target in the form of concise geometric parameters.

[0076] In this embodiment, step 102 includes the following process:

[0077] Step 1021: Process the initial point cloud data to obtain the category probability of each point.

[0078] In step 1021, the category probability refers to the likelihood that a three-dimensional spatial point belongs to any category in a preset category set. The category probability is a multi-dimensional numerical vector. For example, for a category set containing buildings, vegetation, vehicles, and pedestrians, the category probability of a point can be [0.05, 0.02, 0.91, 0.02], where the numerical value represents the probability that the point belongs to a building, vegetation, vehicle, or pedestrian.

[0079] In this embodiment of the application, the initial point cloud data is first input into a pre-trained point cloud understanding model. The model calculates based on the spatial coordinates of the point cloud and its neighborhood relationship. The model outputs a category probability vector for each three-dimensional coordinate point in the point cloud. The value of each component in the category probability vector is obtained by the model internal calculation, representing the confidence level of the point belonging to the corresponding category.

[0080] It should be noted that the embodiments of this application do not impose specific limitations on the model structure and specific functions of the point cloud understanding model, and the specific settings can be made according to the actual situation.

[0081] Step 1022: Based on the category probability, assign each point to the corresponding category and label the category to form initial semantic information.

[0082] In this embodiment of the application, for each point in the point cloud, the probability component with the largest value in the category probability vector of that point is first found, and then the preset category corresponding to that component is determined as the final category of that point, and the point is labeled with the corresponding category label; when all points in the point cloud have completed category assignment and labeling, the original point cloud data together with the category label information of all points constitute the initial semantic information.

[0083] Step 1023: Filter out all points whose category labels belong to the preset dynamic target category from the initial semantic information.

[0084] In step 1023, the preset dynamic target category is a predefined set of categories. The objects corresponding to these categories are usually moving in the scene, such as the "vehicle" and "pedestrian" categories.

[0085] In this embodiment of the application, the categories belonging to the dynamic target category are determined from the category label set according to the preset settings; then, based on the initial semantic information generated in step 1022, all points are traversed, and those points whose category labels match any preset dynamic target category are extracted, and then these points are collected separately.

[0086] Step 1024: Cluster the selected points to identify different dynamic targets.

[0087] In this embodiment of the application, based on the three-dimensional spatial coordinates of all points selected from the initial semantic information, the Euclidean distance between the points is calculated, and finally the points with an Euclidean distance less than a preset threshold and the same category label are grouped into the same set; where each set represents an independent dynamic target individual, such as a vehicle or a pedestrian.

[0088] Step 1025: For each dynamic target, determine the corresponding three-dimensional spatial boundary and generate initial vector data based on the three-dimensional spatial boundary.

[0089] In step 1025, the three-dimensional spatial boundary is used to describe the range occupied by a dynamic target in three-dimensional space, which is usually represented by a three-dimensional geometric bounding box.

[0090] In this embodiment of the application, for each set of dynamic target points identified by clustering, the maximum and minimum values ​​of the set of points in all three-dimensional coordinate directions are first calculated to determine the smallest three-dimensional cuboid bounding box that can completely contain the set of points. This cuboid bounding box is the three-dimensional spatial boundary of the dynamic target. Then, the geometric parameters of the cuboid bounding box, such as its center point coordinates, length, width, height and orientation, are organized into structured data, which is the initial vector data of the dynamic target.

[0091] Through steps 1021 to 1025 above, this application can automatically identify moving objects from the original point cloud, accurately frame and parameterize them, and provide accurate structured input for subsequent targeted data acquisition and refined 3D reconstruction.

[0092] Step 103: Based on the initial vector data and combined with the real-time pose information of the mobile carrier, generate control commands.

[0093] Among them, real-time pose information is used to describe the instantaneous state of the mobile carrier in three-dimensional space, and this information includes position information and orientation information.

[0094] In this embodiment, step 103 includes the following process:

[0095] Step 1031: Obtain the real-time pose information of the mobile carrier, the real-time pose information including the position information and orientation information of the mobile carrier at the current moment.

[0096] In step 1031, the position information is the three-dimensional coordinates of the mobile carrier in the preset world coordinate system, and the orientation information is the orientation angle of the mobile carrier relative to each coordinate axis of the world coordinate system.

[0097] In this embodiment of the application, the motion data of the mobile carrier is continuously collected by the positioning module and the inertial measurement unit integrated on the mobile carrier; wherein the positioning module provides the three-dimensional position coordinates of the carrier in a preset world coordinate system, and the inertial measurement unit provides the attitude angle data of the carrier. These two parts of data are time-aligned to form real-time pose information.

[0098] Step 1032: Determine the center position of the dynamic target in three-dimensional space based on the initial vector data.

[0099] In step 1032, the center position refers to a three-dimensional coordinate point directly extracted from the initial vector data generated in step 102, which represents the center position of the dynamic target in its three-dimensional spatial boundary geometric bounding box.

[0100] In this embodiment, since the initial vector data already contains the geometric bounding box parameters of the dynamic target and explicitly records the coordinates of the center point of the bounding box, this application can obtain the center position of the dynamic target by reading the center point coordinates from the initial vector data.

[0101] Step 1033: Determine the observation direction of the sensor group installed on the mobile carrier based on the position and orientation information of the mobile carrier.

[0102] In step 1033, the sensor group observation pointing is a direction vector in three-dimensional space, which is used to characterize the direction pointed to by the joint observation main axis of the camera and lidar installed on the mobile carrier in the current carrier pose.

[0103] In this embodiment, the rotational transformation relationship from the carrier's own coordinate system to the preset world coordinate system is first determined based on the orientation information of the moving carrier; then, based on the preset fixed installation orientation of the sensor group in the carrier's own coordinate system and the rotational transformation relationship, the observation pointing vector of the sensor group in the world coordinate system is calculated.

[0104] Step 1034: Calculate the angle difference and distance difference between the center position and the observation direction.

[0105] In step 1034, the angle difference refers to the angle in three-dimensional space between the direction of the line connecting the sensor position to the target center position and the direction the sensor is currently observing; the distance difference refers to the straight-line distance from the sensor position to the target center position.

[0106] In this embodiment, firstly, the installation position of the sensor group is calculated by combining the position information of the carrier with the fixed offset of the sensor relative to the carrier. Then, the direction vector from the installation position of the sensor group to the center position of the dynamic target is calculated. Next, the angle between the direction vector and the observation pointing vector is calculated by the vector dot product formula. This angle is the angle difference. At the same time, the distance difference is obtained by calculating the Euclidean distance between the two points from the starting point to the center position.

[0107] It should be noted that the specific calculation methods used for angle difference and distance difference in the embodiments of this application are not specifically limited, and can be set according to the actual situation.

[0108] Step 1035: Based on the angle difference and the distance difference, and in conjunction with the preset servo control logic, generate a servo compensation command.

[0109] In step 1035, the servo compensation command is a specific action command sent to the servo gimbal carrying the sensor, used to drive the gimbal to rotate and change the sensor's observation direction.

[0110] In this embodiment, the preset servo control logic defines the mapping relationship between the angle difference and the distance difference and the required rotation angle of the gimbal; then, based on the calculated angle difference and distance difference, the corresponding control logic is queried and converted into two specific angle values ​​that drive the gimbal to rotate in the horizontal direction and the vertical direction, wherein these two angle values ​​can constitute a servo compensation command.

[0111] In practical applications, the calculated angle difference is 25 degrees and the distance difference is 90.5 meters. The preset control logic mapping table is consulted, which may stipulate that when the distance difference is at the level of 90 meters, the horizontal angle difference must be fully compensated. Based on this, the servo compensation command is generated as "rotate the horizontal axis by 25 degrees and the vertical axis by 0 degrees" to align with the target.

[0112] Step 1036: Generate a path planning instruction pointing to the center position based on the preset scene scanning path.

[0113] In step 1036, the path planning instruction is a drive instruction for the LiDAR scanning mirror, used to plan a fine scanning trajectory of the LiDAR within a small range after the servo gimbal is roughly aligned with the target, so as to cover the dynamic target.

[0114] In this embodiment, there is a preset global scanning path strategy applicable to the large-scale digital twin scenario; then, after obtaining the center position of the dynamic target, a sub-module is extracted from the global strategy. This sub-module defines a reciprocating scanning line pattern with the center position as the focus; then, based on the current target position and the gimbal attitude, the specific step sequence required by the scanning mirror to execute the specific mode is calculated. This sequence is the path planning instruction.

[0115] Step 1037: Combine the servo compensation instruction with the path planning instruction to form a control instruction.

[0116] In step 1037, the control command is a set of multiple sub-instructions used to coordinate the control of the sensor acquisition system. The servo compensation instruction and path planning instruction in this set are used to control different execution components.

[0117] In this embodiment, the servo compensation command generated in step 1035 and the path planning command generated in step 1036 are packaged together to form a structured data packet or command stream; wherein the data packet is sent to the control unit of the sensor acquisition system, which parses it and drives the servo gimbal and the lidar scanning mirror to perform corresponding actions respectively.

[0118] This application achieves the real-time calculation and generation of precise sensor control commands based on the position of the dynamic target and the state of the carrier itself through the above steps, thereby providing accurate action basis for subsequent acquisition of targeted and high-quality data.

[0119] Step 104: Adjust the acquisition parameters according to the control command to obtain the target video stream and target point cloud data for the dynamic target. The acquisition parameters may include a preset duration.

[0120] In this embodiment, step 104 includes the following process:

[0121] Step 1041: Send the servo compensation command in the control command to the sensor platform to drive the sensor platform to rotate so that the sensor observation direction is aligned with the center position.

[0122] In step 1041, the sensor platform refers to the servo gimbal mechanism that carries and physically integrates the camera and lidar. This mechanism can receive electrical signal commands and drive itself to rotate precisely in both the horizontal and vertical axes. The center position is the geometric center point coordinate of the dynamic target in three-dimensional space as determined in step 1032.

[0123] In this embodiment, a servo compensation command specifically for controlling the rotation of the gimbal is parsed and extracted from the control command data packet generated in step 103. This command contains two specific values: a horizontal rotation angle and a vertical rotation angle. Then, the servo compensation command is sent to the motor drive controller of the sensor platform through the communication interface. The motor drive controller generates a corresponding pulse signal based on the received angle value to drive the servo motor to rotate, thereby driving the entire sensor platform to rotate until the observation axis of the sensor on the platform is aligned with the center position of the pre-calculated dynamic target.

[0124] Step 1042: After the sensor platform has rotated into position, the laser radar is controlled to scan and the camera is controlled to acquire images according to the path planning instructions in the control commands.

[0125] In step 1042, "rotation into position" refers to the state where the sensor platform confirms that it has reached the target angle position after completing the rotation action required by the servo compensation command and receiving feedback from its built-in angle encoder; the path planning command is a sub-command in the control command used to finely control the deflection sequence of the lidar scanning mirror.

[0126] In this embodiment, after the sensor platform rotates to its position and sends a positioning signal, a path planning instruction is parsed from the control command. This instruction is sent to the internal scanning control unit of the lidar. The control unit drives the scanning galvanometer or rotating mirror to deflect sequentially to a specified angle according to the scanning point sequence and timing defined in the instruction, thereby performing a fine scan of a specific pattern in the target area. At the same time, an image acquisition trigger signal is sent to the camera to control the camera to start continuously capturing images at a preset frame rate higher than the normal acquisition frame rate, thereby ensuring that the image acquisition and the fine scan of the lidar are synchronized in time.

[0127] Step 1043: While the sensor platform remains aligned, continuously perform data acquisition operations for a preset duration to obtain target video stream and target point cloud data.

[0128] In step 1043, the preset duration is a pre-set time length, such as 0.5 seconds or 1 second, which is sufficient for the lidar to complete at least one round of fine scanning pattern and obtain a sufficiently dense point cloud, while allowing the camera to capture multiple consecutive clear images.

[0129] In this embodiment, while the sensor platform remains locked and aligned with the center of the dynamic target, the lidar continuously executes the fine scanning mode defined by the path planning instruction, and the camera continuously captures images at a high frame rate. This collaborative acquisition process lasts for a complete preset duration. After the acquisition is completed, all continuous image frames acquired by the camera during this period are arranged in chronological order to form a target video stream. At the same time, all three-dimensional points obtained by the lidar during this period are collected to form target point cloud data with a spatial point density higher than the initial point cloud data.

[0130] This application achieves real-time dynamic control and collaborative acquisition of sensors through the above steps, thereby enabling the proactive acquisition of multimodal data with higher spatial density and higher temporal resolution for specific dynamic targets, providing a high-quality data foundation for subsequent refined 3D reconstruction.

[0131] Step 105: Based on the target vector data corresponding to the target video stream and the target point cloud data, and combined with a temporal fusion network, generate fused feature data.

[0132] The target vector data is parameterized data generated from target point cloud data, used to accurately describe the three-dimensional geometry and spatial position of dynamic targets. It should be noted that this application does not specifically limit the structure and function of the temporal fusion network; specific settings can be made according to actual circumstances.

[0133] In this embodiment, step 105 includes the following process:

[0134] Step 1051: Process the target point cloud data to generate target vector data.

[0135] In this embodiment of the application, the same processing flow as that used in step 102 to generate the initial vector data is adopted to process the target point cloud data obtained in step 104. Specifically, the target point cloud data is first semantically segmented to identify dynamic targets, then different target individuals are distinguished by clustering, and finally the three-dimensional spatial boundary of each target individual is calculated and converted into geometric parameters, thereby generating target vector data with higher accuracy and richer details.

[0136] Step 1052: Select keyframes from the target video stream that are time-aligned with the target vector data.

[0137] In step 1052, time alignment refers to selecting the image frame in the target video stream whose timestamp is consistent with or closest to the acquisition time of the target point cloud data on which the target vector data depends.

[0138] In this embodiment of the application, while acquiring target point cloud data, the timestamp of each scanning cycle is recorded; similarly, when acquiring target video stream, each frame image is also timestamped; then, based on the timestamp of the target point cloud data, the frame image with the smallest difference between the timestamp and the target video stream image frame sequence is found, and this frame image is used as a keyframe.

[0139] Step 1053: Input the target vector data and the keyframes into the temporal fusion network. Through the extraction module of the temporal fusion network, extract image features from the video keyframe sequence and extract spatial features from the target vector data.

[0140] In step 1053, the image features are high-dimensional numerical matrices representing the image content and texture extracted from keyframes by the convolutional neural network, and the spatial features are numerical vectors representing the three-dimensional position and size of the target extracted from the geometric parameters of the target vector data.

[0141] In this embodiment, firstly, multiple sets of target vector data that are sequentially continuous in time and corresponding multiple key frames are combined into time series pairs and input into the image branch of the extraction module. This branch consists of multiple layers of convolution and pooling operations, and finally outputs the image feature matrix corresponding to each frame. At the same time, the time series pairs are input into the vector branch of the extraction module. This branch encodes geometric parameters into spatial feature vectors through several fully connected layers.

[0142] Step 1054: The image features are weighted in the image space based on the spatial features through the adjustment module of the temporal fusion network.

[0143] In step 1054, the adjustment module is a sub-network within the temporal fusion network. Its function is to use the three-dimensional positional information contained in the spatial features to guide and reallocate the importance weights of each spatial location on the image feature map.

[0144] In this embodiment, the adjustment module first transforms the spatial feature vector at each time step to generate a two-dimensional weight map that matches the size of the image feature map. Then, the weight map is multiplied element-wise with the image feature map at the corresponding time step to apply different weights to different regions of the image feature map, thereby making the network pay more attention to the image region corresponding to the three-dimensional position of the dynamic target.

[0145] Step 1055: Through the output module of the temporal fusion network, the weighted image features and the spatial features are spliced ​​and transformed along the channel dimension to output fused feature data.

[0146] In step 1055, the output module is the last part of the temporal fusion network, which is responsible for integrating the features of different modalities and outputting data in a unified format; the fused feature data is a high-dimensional numerical representation that integrates multimodal spatiotemporal information.

[0147] In this embodiment, the output module first flattens the image feature map after weight adjustment at each time step into a one-dimensional vector; then, it concatenates this vector with the spatial feature vector at that time step in the channel dimension to form a longer fusion vector; next, it inputs the fusion vectors from multiple time steps into a recurrent neural network layer or a temporal convolutional layer to capture the correlation information in the time dimension; finally, it performs dimensionality reduction and integration of the temporally processed features through a fully connected layer to output the final fusion feature data.

[0148] This application achieves deep fusion of high-precision target 3D vector information and video image information in time and space through the above steps, generating a joint feature representation that can simultaneously characterize the precise 3D geometric attributes and rich appearance texture of dynamic targets, providing comprehensive and highly correlated input for subsequent neural radiation field reconstruction.

[0149] Step 106: Based on the fused feature data, drive the neural radiation field to perform three-dimensional reconstruction and rendering, and generate real-time rendering results.

[0150] In this embodiment, step 106 includes the following process, such as... Figure 2 As shown:

[0151] Step 1061: Input the fused feature data and the target vector data into the neural radiation field, and determine the dynamic target region in the neural radiation field based on the target vector data.

[0152] In step 1061, the neural radiation field is a function used for 3D scene modeling, which maps the coordinates and viewing direction of a 3D point to the color and density value of that point; the dynamic target region is a subspace range in 3D space defined by the geometric bounding box defined by the target vector data.

[0153] It should be noted that the specific expression of the above function is not specifically limited in the embodiments of this application, and can be set accordingly according to the actual situation.

[0154] In this embodiment of the application, the fusion feature data generated in step 105 is used as the conditional input of the neural radiation field, and the target vector data generated in step 1051 is also input. The neural radiation field first analyzes the target vector data and extracts the center coordinates, length, width and height dimensions and orientation of the bounding box, thereby clearly defining a cubic region representing the dynamic target in three-dimensional spatial coordinates. This region is the dynamic target region.

[0155] In practical applications, the fused feature data is a vector of length 1024, and the target vector data includes the center coordinates [20.85, 6.05, 0.72] and dimensions [1.38, 1.55, 2.05]. Then, based on these parameters, the neural radiation field determines a cuboid space in three-dimensional space with its center located at [20.85, 6.05, 0.72] and its length, width, and height being 1.38 meters, 1.55 meters, and 2.05 meters, respectively. The interior of this cuboid is the dynamic target region to be finely reconstructed.

[0156] Step 1062: Perform three-dimensional reconstruction on the sampling points located within and outside the dynamic target area to obtain the reconstruction results.

[0157] The sampling points are a series of three-dimensional spatial points selected according to a certain strategy on an observation ray that originates from the virtual camera and passes through the pixels of the virtual imaging plane. The reconstruction results include the first reconstruction result corresponding to the sampling points located within the dynamic target area and the second reconstruction result corresponding to the sampling points located outside the dynamic target area.

[0158] As a specific implementation, step 1062 may include the following steps:

[0159] A1: Determine the location attributes of each sampling point based on the target vector data.

[0160] In step A1, the location attribute is a binary label used to identify whether a sampling point is located inside or outside the dynamic target area.

[0161] In this embodiment of the application, for each sampling point on each observation ray, its three-dimensional coordinates are read; then the coordinates are compared with the boundary of the dynamic target area defined by the target vector data. If the coordinates are within the boundary range of the three axes of the cuboid, its position attribute is determined to be "within the dynamic target area", otherwise its position attribute is determined to be "outside the dynamic target area".

[0162] In practical applications, for a sampling point on an observation ray, its coordinates are [21.0, 6.1, 0.8]. This point is compared with the boundary of the dynamic target region, which is defined by the X-axis range of 20.16 to 21.54, the Y-axis range of 5.275 to 6.825, and the Z-axis range of -0.305 to 1.745. Since 21.0 is between 20.16 and 21.54, 6.1 is between 5.275 and 6.825, and 0.8 is between -0.305 and 1.745, the location attribute of this sampling point is determined to be "within the dynamic target region".

[0163] A2: For sampling points whose location attribute is within a dynamic target area, query a preset first density table to obtain a first density value, and query a preset first color table to obtain a first color value.

[0164] In step A2, the first density table and the second density table are functions or data structures stored in the neural radiation field for querying spatial point density values, wherein the first density table performs higher resolution modeling for the dynamic target region; the first color table and the second color table are functions or data structures for querying spatial point color values, wherein the first color table associates richer texture information for the dynamic target region; the first density value and the first color value represent the opacity and visual color of the sampling point located within the dynamic target region at that location, respectively.

[0165] In this embodiment of the application, two query mechanisms are maintained within the neural radiation field. For sampling points whose location attribute is "within the dynamic target area", their three-dimensional coordinates and observation direction vector are input into the first query mechanism. This mechanism first searches and interpolates in the first density table to obtain the first density value of the point. Then, in the first color table, it calculates the first color value of the point by combining the fused feature data.

[0166] In practical applications, for a sampling point with coordinates [21.0, 6.1, 0.8] and observation direction [0.1, 0.9, -0.3], since its location attribute is within the region, the first mechanism is used; the first density table with a resolution of 128×128×128 is queried, and the first density value of the point is obtained by trilinear interpolation, which is 0.85; at the same time, the coordinates, observation direction and fused feature data of the point are input into an 8-layer multilayer perceptron, and the first color value of the point is output as [0.75, 0.12, 0.08].

[0167] A3: For sampling points whose location attribute is outside the dynamic target area, query the preset second density table to obtain the second density value, and query the preset second color table to obtain the second color value.

[0168] In step A3, the second density value and the second color value represent the opacity and visual color of the sampling point located outside the dynamic target area at that location, respectively.

[0169] In this embodiment of the application, for a sampling point whose location attribute is "outside the dynamic target area", its three-dimensional coordinates and observation direction vector are input into the second query mechanism; the mechanism searches and interpolates in the second density table to obtain the second density value of the point; then, in the second color table, combined with the fused feature data, the second color value of the point is calculated.

[0170] In practical applications, for a sampling point with coordinates [50.0, 30.0, 10.0] and observation direction [0.0, 0.0, -1.0] on another ray, the second mechanism is used because its location attribute is outside the region; then, the second density table with a resolution of 64×64×64 is queried, and the second density value of the point is obtained by trilinear interpolation, which is 0.02; at the same time, the coordinates, observation direction and fused feature data of the point are input into a 4-layer multilayer perceptron, and the second color value of the point is finally output as [0.4, 0.5, 0.6].

[0171] A4: Calculate the first density value and the first color value along the preset observation ray to generate the first reconstruction result.

[0172] In step A4, the first reconstruction result is the color value of the corresponding pixel in the rendered two-dimensional image, which is the result of an observation ray passing through the scene, considering only the contribution of sampling points located within the dynamic target area.

[0173] In this embodiment of the application, for an observation ray, firstly, all sampling points on the ray with the position attribute of "dynamic target area" are selected; then, according to the order of these sampling points along the ray from near to far, the volume rendering integral formula is used to accumulate and calculate the contribution of the first density value and the first color value of these sampling points to the final output color of the ray; the calculated color value is the first reconstruction result of the dynamic target part corresponding to the pixel.

[0174] It should be noted that the specific expression of the volume rendering integral formula is not specifically limited in the embodiments of this application, and can be set accordingly according to the actual situation.

[0175] In practical applications, for a specific observation ray, five sampling points are identified as being within the "dynamic target area." Along the ray's sequence, the first density values ​​of these five points are 0.1, 0.3, 0.85, 0.4, and 0.05, respectively, and their first color values ​​are... Next, through the volume rendering formula Perform calculations, where The color value calculated by volume rendering. It is the first Cumulative transmittance at each sampling point It is the first The opacity is calculated from the density value of each sampling point. Indicates the first The first color value of each sampling point This represents the total number of sampling points within the "dynamic target area" on the observed ray; a color value is ultimately calculated. For example, [0.70, 0.10, 0.05], this color value This is the first reconstruction result corresponding to the pixel, representing the color of the dynamic target part.

[0176] A5: Calculate the second density value and the second color value along the observed light ray to generate a second reconstruction result.

[0177] In step A5, the second reconstruction result is the color value of the corresponding pixel in the rendered two-dimensional image, which is the result of an observation ray passing through the scene, considering only the contribution of sampling points located outside the dynamic target area.

[0178] In this embodiment, for the same observation ray, all sampling points with the position attribute "outside the dynamic target area" are selected; then, the volume rendering integral formula is used to accumulate and calculate the contribution of the second density value and the second color value of these sampling points to the final output color of the ray; the calculated color value is the second reconstruction result of the static environment part corresponding to the pixel.

[0179] The reconstruction step is the core process for generating three-dimensional geometric and appearance information. This step decodes the density and color distribution of three-dimensional space from the fused feature data. This is the premise for subsequent two-dimensional image rendering to correctly generate pixels based on any viewpoint. Without three-dimensional reconstruction, it is impossible to flexibly and correctly synthesize dynamic and static image regions according to different observation viewpoints, and it is also impossible to ensure the geometric consistency and visual realism of the synthesized image in three-dimensional space.

[0180] In practical applications, on the same observation ray, in addition to the above 5 points, the remaining 45 sampling points are determined to be "outside the dynamic target area". Then, using the same volume rendering formula, the second density value and the second color value of these 45 points are accumulated and calculated to finally obtain a color value Cs, for example [0.35, 0.40, 0.45]. This color value Cs is the second reconstruction result corresponding to the pixel, representing the color of the static background part.

[0181] Step 1063: Based on the reconstruction results, render the dynamic image region and the static image region respectively.

[0182] In step 1063, the dynamic image region is a two-dimensional image block composed of the first reconstruction results of all observed rays arranged in the pixel arrangement of the imaging plane, and the image block contains only the visual content of the dynamic target; the static image region is a two-dimensional image block composed of the second reconstruction results of all observed rays, and the image block contains only the visual content of the static environment.

[0183] In this embodiment of the application, after the calculation of all observed rays is completed, the first reconstruction result Cd of all rays is filled into the corresponding pixel position in the imaging plane to form a complete dynamic image area; at the same time, the second reconstruction result Cs of all rays is filled into the corresponding pixel position in the same imaging plane to form a complete static image area.

[0184] In practical applications, for a virtual imaging plane with a resolution of 1920×1080, it is first necessary to calculate the observation ray corresponding to each pixel, which involves a total of approximately 2 million rays. Then, the first reconstruction result Cd calculated for each observation ray is filled into the corresponding pixel position in a blank image matrix of size 1920×1080, thereby generating a dynamic image region that only displays the car target. At the same time, the second reconstruction result Cs calculated for each observation ray is filled into another blank image matrix of the same size in the same way, thereby generating a static image region that only contains roads and buildings.

[0185] Step 1064: Perform pixel-level synthesis of the dynamic image region and the static image region to generate a real-time rendering result.

[0186] In step 1064, pixel-level compositing refers to the process of mixing two images of identical size according to the color values ​​corresponding to each pixel position to generate the final image; the real-time rendering result is the final output, a complete two-dimensional image containing dynamic targets and static environment.

[0187] In this embodiment, the dynamic image region and the static image region generated in step 1063 are aligned. After alignment, for each pixel position in the image, the color value of the dynamic image region at that position is checked. If the color value is valid, that is, the light corresponding to the pixel passes through the dynamic target, the color of the dynamic image region is used as the final pixel color. Otherwise, the color of the static image region at that position is used as the final pixel color. By traversing all pixels, the synthesis is completed, and the final real-time rendering result image is output.

[0188] In practical applications, during pixel-level compositing, the generated dynamic image region, such as a car image, is first spatially aligned and superimposed with a static image region, such as a background image. Next, a specific pixel in the image is processed; for example, the pixel at coordinates 960 and 540 in the center region corresponds to a valid color value [0.70, 0.10, 0.05] in the dynamic image region. Therefore, the final rendering result at this location directly uses this dynamic color value [0.70, 0.10, 0.05]. Then, for another pixel in the image, such as the pixel at coordinates 10 and 10 on the edge, since the dynamic image region does not provide valid color information at this point, it appears as black or transparent. Therefore, the color value [0.35, 0.40, 0.45] corresponding to this point in the static image region is used as the final color for that location. This process is repeated for each pixel in the imaging plane, and after color selection and filling are completed one by one, a visually complete scene image of a car placed against a road background is synthesized. This image is the final real-time rendering result.

[0189] This application utilizes target vector data to spatially guide the reconstruction process through the above steps, achieving differentiation, refined modeling and rendering of dynamic target areas and static environment areas, and finally synthesizing them into a visually coherent and detailed real-time 3D scene image.

[0190] Figure 3 This application provides a schematic diagram of the structure of a video-driven vector data 3D generation and real-time rendering system based on a digital twin scene, as shown in the embodiments of this application. Figure 3 As shown, the system includes:

[0191] The acquisition module 31 is used to acquire the initial video stream and initial point cloud data of the mobile carrier in a digital twin scenario.

[0192] The segmentation module 32 is used to perform semantic segmentation on the initial point cloud data, generate initial semantic information, and generate initial vector data of the dynamic target based on the category label of the dynamic target in the initial semantic information.

[0193] The first generation module 33 is used to generate control commands based on the initial vector data and the real-time pose information of the mobile carrier.

[0194] The adjustment module 34 is used to adjust the acquisition parameters according to the control command in order to obtain the target video stream and target point cloud data for the dynamic target.

[0195] The second generation module 35 is used to generate fused feature data based on the target vector data corresponding to the target video stream and the target point cloud data, combined with a temporal fusion network.

[0196] The driving module 36 is used to drive the neural radiation field to perform three-dimensional reconstruction and rendering based on the fused feature data, and generate real-time rendering results.

[0197] The video-driven vector data 3D generation and real-time rendering system based on digital twin scenes in this application embodiment is used to implement the aforementioned video-driven vector data 3D generation and real-time rendering method based on digital twin scenes. Therefore, the specific implementation of the video-driven vector data 3D generation and real-time rendering system based on digital twin scenes can be found in the embodiment section of the video-driven vector data 3D generation and real-time rendering method based on digital twin scenes above. The specific implementation can be referred to the description of the corresponding embodiments, and will not be repeated here.

[0198] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of the video-driven vector data 3D generation and real-time rendering method based on digital twin scenes described above.

[0199] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the above-described methods for video-driven vector data 3D generation and real-time rendering based on digital twin scenes.

[0200] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory, random access memory, portable hard drives, magnetic disks, or optical disks.

[0201] The embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above embodiments of the video-driven vector data 3D generation and real-time rendering method based on a digital twin scene.

[0202] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0203] The foregoing has provided a detailed description of a method and system for video-driven vector data 3D generation and real-time rendering based on a digital twin scene, as provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A method for video-driven 3D generation and real-time rendering of video-driven vector data based on digital twin scenarios, characterized in that, include: In a digital twin scenario, the initial video stream and initial point cloud data of the mobile carrier are collected; The initial point cloud data is semantically segmented to generate initial semantic information, and initial vector data of the dynamic targets is generated based on the category labels of the dynamic targets in the initial semantic information. Based on the initial vector data and combined with the real-time pose information of the mobile carrier, control commands are generated. According to the control command, the acquisition parameters are adjusted to obtain the target video stream and target point cloud data for the dynamic target; Based on the target vector data corresponding to the target video stream and the target point cloud data, and combined with a temporal fusion network, fused feature data is generated. Based on the fused feature data, the driving neural radiation field is reconstructed and rendered in three dimensions to generate real-time rendering results. The process of driving the neural radiation field to perform three-dimensional reconstruction and rendering based on the fused feature data, generating real-time rendering results, includes: The fused feature data and the target vector data are input into the neural radiation field, and a dynamic target region is determined in the neural radiation field based on the target vector data. Three-dimensional reconstruction was performed on sampling points located within and outside the dynamic target area to obtain the reconstruction results; Based on the reconstruction results, dynamic image regions and static image regions are rendered respectively; The dynamic image region and the static image region are combined at the pixel level to generate a real-time rendering result; The step of generating control commands based on the initial vector data and the real-time pose information of the mobile carrier includes: The real-time pose information of the mobile carrier is obtained, and the real-time pose information includes the position and orientation information of the mobile carrier at the current moment; Based on the initial vector data, determine the center position of the dynamic target in three-dimensional space; Based on the position and orientation information of the mobile carrier, the observation direction of the sensor group installed on the mobile carrier is determined; Calculate the angular difference and distance difference between the center position and the observation direction; Based on the angle difference and the distance difference, and combined with the preset servo control logic, a servo compensation command is generated. A path planning instruction pointing to the center location is generated based on a preset scene scanning path. The servo compensation instruction and the path planning instruction are combined to form a control instruction; The step of performing semantic segmentation on the initial point cloud data to generate initial semantic information, and generating initial vector data of the dynamic targets based on the category labels of the dynamic targets in the initial semantic information, includes: The initial point cloud data is processed to obtain the category probability of each point; Based on the category probabilities, each point is assigned to a corresponding category and labeled with a category tag to form initial semantic information; Filter out all points whose category labels belong to the preset dynamic target category from the initial semantic information; Cluster the selected points to identify different dynamic targets; For each dynamic target, the corresponding three-dimensional spatial boundary is determined, and initial vector data is generated based on the three-dimensional spatial boundary.

2. The method according to claim 1, characterized in that, The process involves performing three-dimensional reconstruction on sampling points located within and outside the dynamic target area to obtain reconstruction results, including: Based on the target vector data, determine the location attributes of each sampling point; For sampling points whose location attribute is a dynamic target area, a first density value is obtained by querying a preset first density table, and a first color value is obtained by querying a preset first color table. For sampling points whose location attribute is outside the dynamic target area, a second density value is obtained by querying a preset second density table, and a second color value is obtained by querying a preset second color table. Along a preset observation ray, the first density value and the first color value are calculated to generate a first reconstruction result; The second density value and the second color value are calculated along the observed light ray to generate a second reconstruction result.

3. The method according to claim 1, characterized in that, The step of generating fused feature data based on the target vector data corresponding to the target video stream and the target point cloud data, combined with a temporal fusion network, includes: The target point cloud data is processed to generate target vector data; Select keyframes that are time-aligned with the target vector data from the target video stream; The target vector data and the keyframes are input into a temporal fusion network. The extraction module of the temporal fusion network extracts image features from the video keyframe sequence and spatial features from the target vector data. The adjustment module of the temporal fusion network adjusts the weights of the image features in the image space based on the spatial features. The output module of the temporal fusion network concatenates and transforms the weighted image features and spatial features along the channel dimension to output fused feature data.

4. The method according to claim 1, characterized in that, The step of adjusting the acquisition parameters according to the control command to obtain the target video stream and target point cloud data for the dynamic target includes: The servo compensation command in the control command is sent to the sensor platform to drive the sensor platform to rotate so that the sensor observation direction is aligned with the center position. After the sensor platform rotates into position, the laser radar is controlled to scan and the camera is controlled to acquire images according to the path planning instructions in the control commands. While the sensor platform remains aligned, it continuously performs data acquisition operations for a preset duration to obtain target video streams and target point cloud data.

5. A video-driven vector data 3D generation and real-time rendering system based on digital twin scenes, characterized in that, include: The acquisition module is used to acquire the initial video stream and initial point cloud data of the mobile carrier in a digital twin scenario. The segmentation module is used to perform semantic segmentation on the initial point cloud data, generate initial semantic information, and generate initial vector data of the dynamic targets based on the category labels of the dynamic targets in the initial semantic information. The first generation module is used to generate control commands based on the initial vector data and the real-time pose information of the mobile carrier. The adjustment module is used to adjust the acquisition parameters according to the control command in order to obtain the target video stream and target point cloud data for the dynamic target; The second generation module is used to generate fused feature data based on the target vector data corresponding to the target video stream and the target point cloud data, combined with a temporal fusion network. The driving module is used to drive the neural radiation field to perform three-dimensional reconstruction and rendering based on the fused feature data, and generate real-time rendering results; The process of driving the neural radiation field to perform three-dimensional reconstruction and rendering based on the fused feature data, generating real-time rendering results, includes: The fused feature data and the target vector data are input into the neural radiation field, and a dynamic target region is determined in the neural radiation field based on the target vector data. Three-dimensional reconstruction was performed on sampling points located within and outside the dynamic target area to obtain the reconstruction results; Based on the reconstruction results, dynamic image regions and static image regions are rendered respectively; The dynamic image region and the static image region are combined at the pixel level to generate a real-time rendering result; The step of generating control commands based on the initial vector data and the real-time pose information of the mobile carrier includes: The real-time pose information of the mobile carrier is obtained, and the real-time pose information includes the position and orientation information of the mobile carrier at the current moment; Based on the initial vector data, determine the center position of the dynamic target in three-dimensional space; Based on the position and orientation information of the mobile carrier, the observation direction of the sensor group installed on the mobile carrier is determined; Calculate the angular difference and distance difference between the center position and the observation direction; Based on the angle difference and the distance difference, and combined with the preset servo control logic, a servo compensation command is generated. A path planning instruction pointing to the center location is generated based on a preset scene scanning path. The servo compensation instruction and the path planning instruction are combined to form a control instruction; The step of performing semantic segmentation on the initial point cloud data to generate initial semantic information, and generating initial vector data of the dynamic targets based on the category labels of the dynamic targets in the initial semantic information, includes: The initial point cloud data is processed to obtain the category probability of each point; Based on the category probabilities, each point is assigned to a corresponding category and labeled with a category tag to form initial semantic information; Filter out all points whose category labels belong to the preset dynamic target category from the initial semantic information; Cluster the selected points to identify different dynamic targets; For each dynamic target, the corresponding three-dimensional spatial boundary is determined, and initial vector data is generated based on the three-dimensional spatial boundary.

6. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the steps of the video-driven vector data 3D generation and real-time rendering method based on a digital twin scene as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, enables the implementation of the video-driven vector data 3D generation and real-time rendering method based on a digital twin scene as described in any one of claims 1 to 4.

Images