Video picture and three-dimensional modeling fusion method and system based on digital twin scene

Abstract

The present disclosure provides a video picture and three-dimensional modeling fusion method and system based on a digital twin scene, which comprises the following steps: instance generation, visual field blind area prediction, cooperative tracking, blind area crossing processing and neighborhood expansion processing. By listening to the speed pass gate card swiping event as the real-time tracking trigger condition of the employee, combining the corresponding employee identity and the real-time position update based on the camera, the real-time rendering of the employee position in the digital twin model can be completed without additional device deployment, improving the overall management effect of the park on employees; and the next camera is predicted through the camera topological relationship to avoid the waste of calculation caused by blind search in the whole area; finally, the identity authentication of the entering personnel is realized within a reasonable time range, the blind area crossing is realized, the interruption of target employee tracking under non-continuous coverage is solved, and the deep fusion of video and twin model is realized.

Description

Technical Field

[0001] This disclosure relates to the field of digital twin technology, and in particular to a method and system for fusing video footage and 3D modeling based on a digital twin scene, as well as a storage medium and electronic device. Background Technology

[0002] In the field of smart park management, digital twin technology is widely used to construct virtual mappings of physical parks to achieve visualized real-time situational awareness. Current mainstream solutions for overlaying personnel location information into digital twin models typically rely on additional positioning devices such as UWB tags, Bluetooth, or RFID cards worn by personnel. Coordinates are obtained through positioning base stations and then displayed as static points in the model. However, these solutions not only require high hardware deployment costs and continuous tag maintenance costs, but their positioning accuracy is also significantly affected by signal obstruction. More importantly, the personnel locations presented in the digital twin model are often only discrete points or the last known location, failing to realistically and continuously reflect the real-time movement of personnel between park roads, plazas, and buildings. This results in a low degree of virtual-real integration, making it difficult to meet the real-time situational awareness requirements of park security, emergency dispatch, and other scenarios.

[0003] On the other hand, while the numerous video surveillance cameras deployed within the park can capture real-time footage of people's activities, the current integration of video analytics and digital twin systems is rather rudimentary. It can only render the digital twin model based on texture mapping of the surveillance footage, failing to provide real-time feedback on personnel movement within the park and consuming significant computing resources. In real-world park environments, blind spots are prevalent due to factors such as building obstructions, green space barriers, and camera installation angle limitations. Existing technologies cannot effectively address the continuous positioning of personnel during transitions within these blind spots, leading to frequent instances of frozen positions, disappearances, or erroneous jumps in the digital twin model. This severely restricts the practical value of digital twin systems in smart parks. Summary of the Invention

[0004] The purpose of this disclosure is to provide a method and system, storage medium, and electronic device for fusing video footage and 3D modeling based on a digital twin scene, in order to solve the problems existing in the prior art.

[0005] The embodiments of this disclosure adopt the following technical solution: a method for fusing video footage and 3D modeling based on digital twin scenes, applied to smart parks with video surveillance, including: Instance generation steps: Listen for the first card swipe event at the speed gate at the park entrance, determine the identity information and initial model position of the target employee based on the trigger time and card ID of the first card swipe event, create a personnel instance corresponding to the target employee, and render the personnel instance in real time at the initial model position in the digital twin model of the park, and set the camera that captures the initial model position as the current camera. Blind spot prediction steps: Based on the topology of the cameras in the park, determine the next camera of the current camera, and determine whether there is a blind spot between the next camera and the current camera; Collaborative tracking steps: If there is no blind spot between the next camera and the current camera, drive the current camera and the next camera to track the target employee, update the real-time model position of the target employee, update the rendering of the person instance in the digital twin model according to the real-time model position, set the next camera as the new current camera and return to the blind spot prediction step. Blind Spot Crossing Processing Steps: In the event of a blind spot between the next camera and the current camera, the current camera is driven to track the target employee and update the real-time model position of the target employee. A reasonable time range is calculated based on the time it takes for the target employee to disappear from the current camera's view. Within the reasonable time range, the identity of the person entering the next camera is verified. If the verification is successful, the next camera is driven to track the target employee and update the real-time model position of the target employee. The rendering of the person instance in the digital twin model is updated based on the real-time model position. The next camera is then designated as the new current camera, and the process returns to the blind spot prediction step. Neighborhood expansion processing steps: If no personnel enter the next camera within the reasonable time range or authentication fails, identify all target cameras adjacent to the current camera's field of view. Perform identity authentication within the field of view of each target camera. If the target employee appears within the field of view of any target camera and authentication passes, designate that target camera as the new current camera, track the target employee, update the real-time model position of the target employee, and return to the blind spot prediction step. If the target employee does not appear within the field of view of any target camera or authentication fails, mark the personnel instance as an abnormal state in the digital twin model.

[0006] This disclosure also provides a system for fusing video footage and 3D modeling based on a digital twin scene, applicable to smart parks with video surveillance, including: The instance generation module is used to listen to the first card swipe event at the speed gate at the park entrance, determine the identity information and initial model position of the target employee based on the trigger time and card ID of the first card swipe event, create a personnel instance corresponding to the target employee, and render the personnel instance in real time at the initial model position in the digital twin model of the park, and set the camera that captures the initial model position as the current camera. The blind spot prediction module is used to determine the next camera of the current camera based on the topology of the cameras in the park, and to determine whether there is a blind spot between the next camera and the current camera. The collaborative tracking module is used to drive the current camera and the next camera to track the target employee when there is no blind spot between the next camera and the current camera, update the real-time model position of the target employee, update the rendering of the person instance in the digital twin model according to the real-time model position, and use the next camera as the new current camera. The blind spot crossing processing module is used to drive the current camera to track the target employee and update the real-time model position of the target employee when there is a blind spot between the next camera and the current camera. It calculates a reasonable time range based on the time it takes for the target employee to disappear from the current camera's view, and performs identity authentication on the person entering the next camera within the reasonable time range. When the authentication is successful, it drives the next camera to track the target employee and update the real-time model position of the target employee. Based on the real-time model position, it updates the rendering of the person instance in the digital twin model and sets the next camera as the new current camera. The neighborhood expansion processing module is used to determine all target cameras adjacent to the current camera's field of view if no personnel enter the next camera within the reasonable time range or if authentication fails. Identity authentication is performed within the field of view of each target camera. If the target employee appears within the field of view of any target camera and authentication is successful, that target camera is designated as the new current camera. Target tracking is performed on the target employee, and the real-time model position of the target employee is updated. The process then returns to the blind spot prediction step. If the target employee does not appear within the field of view of any target camera or authentication fails, the personnel instance is marked as abnormal in the digital twin model.

[0007] This disclosure also provides a storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described method for fusing video footage and 3D modeling based on a digital twin scene.

[0008] This disclosure also provides an electronic device, including at least a memory and a processor. The memory stores a computer program, and when the processor executes the computer program in the memory, it implements the steps of the above-described method for fusing video footage and 3D modeling based on a digital twin scene.

[0009] The beneficial effects of this disclosed embodiment are as follows: by listening to the card swiping event of the speed gate as the trigger condition for real-time employee tracking, combined with the corresponding employee identity and real-time location updates based on the camera, the real-time rendering of the employee's location in the digital twin model can be completed without the deployment of additional equipment, thereby improving the overall management effect of the park on employees; and by predicting the next camera through the camera topology relationship, the waste of computation caused by blindly searching the entire area can be avoided; finally, by combining a reasonable time range, the identity of the person entering is authenticated within the window, thereby achieving blind spot crossing, solving the interruption of target employee tracking under non-continuous coverage, and realizing the deep integration of video and the twin model. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in one or more embodiments of this specification or in the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a flowchart of the method for fusing video footage and 3D modeling based on a digital twin scene in the first embodiment of this disclosure; Figure 2 This is a schematic diagram of the structure of the video footage and 3D modeling fusion system based on a digital twin scene in the second embodiment of this disclosure. Detailed Implementation

[0012] To enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of this document.

[0013] To address the problems existing in the prior art, the first embodiment of this disclosure provides a method for fusing video footage and 3D modeling based on a digital twin scene. This method is mainly applied to smart parks with video surveillance. The park has deployed multiple high-definition network cameras at key locations such as main entrances and exits, main roads, and building entrances, and has established a digital twin model that corresponds to the physical park in a 1:1 ratio, thereby achieving centralized real-time presentation of the park's status.

[0014] Before implementing the method provided in this embodiment, system initialization configuration is required. The system in this embodiment mainly includes a digital twin platform, a video surveillance network, a park entrance / exit speed gate system, a face recognition and target tracking server, and a trajectory storage database to support the implementation of the method. Specifically, a digital twin model of the park is first established. A digital twin model is a virtual model obtained by high-precision digital reconstruction of the physical park using 3D modeling technology. Its geometric data can originate from park BIM (Building Information Modeling) data, 3D laser scanning point cloud data, or manually modeled data. The digital twin model includes not only 3D geometric models of park buildings, roads, and green landscapes, but also 3D models of park access paths. These access paths include main roads, branch roads, pedestrian paths, connecting corridors, and other passable areas. These paths are marked in the model as line segments or curved surfaces and associated with attributes such as direction of travel, width, and slope.

[0015] Subsequently, the spatial location, shooting range, and orientation angle of each camera are marked in the digital twin model to establish the topological relationship of the cameras in the park, describing the spatial association between the cameras. This not only records the three-dimensional installation coordinates of each camera in the model coordinate system, but also records the intrinsic parameters (such as focal length and field of view) and extrinsic parameters (such as pitch angle, yaw angle, and roll angle) of each camera, thereby enabling accurate calculation of the shooting range of each camera on the real ground. In some embodiments, each camera can be used as a graph node. If the shooting ranges of two cameras have a boundary or overlap, an edge is established between the corresponding two nodes, and weight attributes such as spatial distance and accessibility of the travel path can be marked on the edge to form a topological graph.

[0016] Finally, each camera is calibrated to establish a mapping transformation relationship between the pixel coordinate system of each camera and the model coordinate system of the digital twin model. The pixel coordinate system is the two-dimensional image coordinate system captured by the camera, while the model coordinate system is the three-dimensional world coordinate system used by the digital twin model. Through the camera calibration process, a projection transformation relationship from pixel coordinates to model coordinates is established. It should be noted that the mapping transformation relationship between the pixel coordinate system and the model coordinate system of different cameras may change over time due to factors such as temperature changes and equipment aging. To avoid these changes affecting the subsequent positioning effect, the mapping transformation relationship of the cameras can be recalibrated at specific intervals.

[0017] Figure 1 The flowchart of the fusion method in this embodiment is shown, mainly including steps S10 to S50: S10, Instance Generation Steps: Listen for the first card swipe event at the speed gate at the park entrance. Determine the identity information of the target employee and the initial model position based on the trigger time and card ID of the first card swipe event. Create a personnel instance corresponding to the target employee and render the personnel instance in real time at the initial model position in the digital twin model of the park. Set the camera that captured the initial model position as the current camera.

[0018] In this embodiment, the system continuously monitors the first card swipe event at the speed gate at the park entrance. An IC card reader or NFC card reader is installed at the speed gate. When a target employee approaches with their card and completes the swipe, the reader generates a trigger signal. The system captures the trigger time and card ID of this first card swipe event. Based on the trigger time and card ID, the system queries the park's personnel management database or access control database to determine the target employee's identity information. This identity information includes at least the employee's name, employee number, department, pre-stored facial template, and permissions. Simultaneously, since the speed gate has pre-calibrated model coordinates in the digital twin model, these model coordinates represent the target employee's initial model position. Alternatively, it can be based on an image of the target employee captured by a camera installed at the speed gate. A target detection algorithm is used to determine the employee's pixel coordinates in the image, and then the initial model position in the digital twin model is determined based on the corresponding mapping transformation relationship.

[0019] After determining the employee's identity and the initial model location, a personnel instance corresponding to the target employee can be created. This is a virtual object in the digital twin model used to represent the real employee. It typically includes a 3D visualization model (such as a simplified human icon, an avatar model, or a cursor with a name tag) and a set of attribute data (including identity information, real-time model location, status markers, etc.). The system renders this personnel instance in real-time at the initial model location in the digital twin model; that is, it loads and displays the virtual object at the corresponding coordinate point in the 3D scene, making it visible in the monitoring interface of the digital twin model. Simultaneously, the camera that captured the initial model location is set as the current camera. The current camera is a status variable used by the system in the tracking process to represent the camera currently tracking the target employee; its initial value is the camera number or identifier covering the speed gate area.

[0020] S20, Blind Spot Prediction Step: Based on the topology of the cameras in the park, determine the next camera of the current camera and determine whether there is a blind spot between the next camera and the current camera.

[0021] After completing instance generation and setting the current camera, the system enters the main loop tracking process, first performing a blind spot prediction step. The core of this step lies in determining the next camera based on the campus camera topology and judging whether a blind spot exists between them. Specifically, the system queries the pre-stored campus camera topology to determine at least one candidate camera spatially adjacent to the current camera. In this embodiment, the candidate camera selection logic can be as follows: search the topology graph for adjacent nodes directly connected to the current camera node, where the camera's shooting range is geographically adjacent to or near the current camera's shooting range. Simultaneously, based on the travel paths in the digital twin model, it determines whether a reachable path exists from the current camera area to the candidate camera area.

[0022] Subsequently, based on the target employee's identity information, the system queries their corresponding historical movement trajectory. This trajectory, recorded and stored by the system during the employee's previous visits to the park using the method described in this embodiment, reflects the employee's movement habits and path preferences within the park. Therefore, this embodiment combines the recorded historical movement trajectory with the passable paths recorded in the digital twin model to determine the next camera from all candidate cameras. In some embodiments, the system can analyze the frequency distribution of each path segment originating from the current area in the historical movement trajectory, and, combined with factors such as the current time and destination probability, select the candidate camera with the highest coverage probability as the next camera. For example, if the current location is at a fork in the park's main road, and the candidate cameras include camera A covering the western branch road and camera B covering the eastern branch road, and the historical movement trajectory shows that the employee has a 90% probability of choosing the eastern branch road, then camera B is determined as the next camera. If the path probabilities of multiple candidate cameras are similar, auxiliary selection can be made based on the camera's shooting range clarity, image resolution, or preset priority rules (such as main road priority).

[0023] Once the next camera is determined, it can be judged whether there is a blind spot or an overlapping area between the two cameras. If the shooting ranges of the two cameras overlap (i.e., the same physical area can be captured by both cameras simultaneously), or their boundaries are directly adjacent (without gaps), then it is determined that there is no blind spot between the current camera and the next camera. If there is an uncovered gap between the two shooting ranges (i.e., there is physical space outside the shooting range of both cameras simultaneously), then it is determined that there is a blind spot.

[0024] S30, Collaborative Tracking Steps: If there is no blind spot between the next camera and the current camera, drive the current camera and the next camera to track the target employee, update the real-time model position of the target employee, update the rendering of the person instance in the digital twin model according to the real-time model position, set the next camera as the new current camera and return to S20.

[0025] When the blind spot prediction step determines that there is no blind spot between the current camera and the next camera, the system executes a collaborative tracking step, driving both the current and next cameras to track the target employee, thereby achieving continuous tracking and real-time location updates. In this embodiment, target tracking refers to the technical process of continuously locating and locking a specific target in a video sequence using computer vision algorithms. This embodiment does not specifically limit the implementation algorithm; the system can run target detection algorithms in the video streams of the current and next cameras respectively, and combine this with the target employee's identity information to achieve real-time tracking and locking of the target employee in the monitoring screen, completing real-time model position updates.

[0026] In this embodiment, the real-time model position refers to the three-dimensional spatial coordinates of the target employee in the digital twin model at the current moment. Specifically, the system extracts the current pixel coordinates of the target employee in the current frame from the real-time video stream captured by the current camera. These current pixel coordinates are typically taken as the coordinates of the bottom center point of the target employee's detection box in the image. This point approximately corresponds to the projection of the employee's feet on the ground. Based on the mapping and transformation relationship between the pixel coordinate system and the model coordinate system established in the initialization preparation steps, the current pixel coordinates are converted into model coordinates within the digital twin model. The Z coordinate is usually taken as the ground height value, and this model coordinate is the real-time model position of the target employee.

[0027] To ensure smooth movement of personnel instances within the digital twin model and avoid abrupt changes in image quality, the system employs a smooth interpolation movement strategy when updating the rendering of personnel instances in the digital twin model based on their real-time model positions. Smooth interpolation refers to calculating intermediate transition positions between adjacent rendering frames instead of directly teleporting the personnel instance from the previous position to the current position. Specifically, the system performs linear or cubic Bézier curve interpolation between the model coordinates of the personnel instance in the previous frame and the real-time model coordinates in the current frame, according to a preset rendering frame rate (e.g., 30 or 60 frames per second), to calculate intermediate transition coordinates, allowing the personnel instance to move smoothly along the ground within the model. Simultaneously, when two cameras simultaneously detect the target employee and both output their real-time model positions, the system weights and fuses the two model coordinates based on the detection confidence of the two cameras and the error weights of the pixel-to-model coordinate conversion, obtaining a unique real-time model position. After updating the rendering, the system uses the next camera as the new current camera and returns to the blind spot prediction step to continue the next tracking loop.

[0028] S40, Blind Spot Crossing Processing Steps: In the event of a blind spot between the next camera and the current camera, drive the current camera to track the target employee and update the real-time model position of the target employee. Calculate a reasonable time range based on the time it takes for the target employee to disappear from the current camera's view. Within the reasonable time range, authenticate the person entering the next camera. If authentication is successful, drive the next camera to track the target employee and update the real-time model position of the target employee. Update the rendering of the person instance in the digital twin model based on the real-time model position. Set the next camera as the new current camera and return to S20.

[0029] When the blind spot prediction step determines that a blind spot exists between the current camera and the next camera, the system executes a blind spot crossing process. At this point, the current camera continues to track the target employee and update the real-time model position until the target employee leaves the current camera's field of view. The system records the time it takes for the target employee to disappear from the current camera's view as a benchmark for calculating the subsequent reasonable time range. The reasonable time range refers to the time window during which the system estimates the target employee's disappearance from the current camera's field of view, passage through the blind spot, and reappearance within the field of view of the next camera.

[0030] In some embodiments, the system first calculates the employee's average walking speed based on the target employee's historical movement trajectory. For example, it extracts all historical trajectory data of the employee within the park over a recent period (e.g., the last 30 days), calculates the walking speed for each trajectory segment, and takes the median or weighted average as the employee's personalized average walking speed. Then, based on the digital twin model and the park's camera topology, it determines the spatial distance between the current camera and the next camera, i.e., the shortest reachable path length along the park's passageway. Next, based on the average walking speed and spatial distance, it predicts the walking time range for the target employee to traverse blind spots. Considering that the employee may accelerate, decelerate, or briefly pause in the blind spot, the system calculates a base time value and adds a preset tolerance deviation (e.g., ±20% or a fixed number of seconds) to obtain the walking time range. Finally, it adds this walking time range to the disappearance time to obtain a reasonable time range. For example, if the disappearance time is... The walking time range is The reasonable time range is .

[0031] Within a reasonable timeframe, the system authenticates the identity of individuals entering the next camera's field of view. Identity authentication refers to the technical process of confirming the identity of individuals in a video through biometric or physical feature comparison. A person re-identification (Re-ID) algorithm can be used to extract the physical feature vectors of individuals entering the next camera's field of view. Similarity calculations are performed between this vector and the target employee's historical physical features to achieve a coarse match. For individuals whose coarse match scores exceed a threshold, their facial images are further extracted and compared with the target employee's pre-stored facial template to achieve precise authentication. Upon successful authentication, it indicates that the target employee has successfully crossed the blind spot. At this point, the system drives the next camera to track the target employee and updates the target employee's real-time model position. The real-time model position update method is consistent with the aforementioned collaborative tracking steps: extracting the target employee's current pixel coordinates in the next camera's view, converting them to model coordinates through a mapping transformation relationship, and using a smooth interpolation movement strategy to update the rendering of the person instance in the digital twin model. Subsequently, the next camera is used as the new current camera, and the system returns to the blind spot prediction step.

[0032] S50, Neighborhood Expansion Processing Steps: If no personnel enter the next camera within a reasonable timeframe or authentication fails, identify all target cameras adjacent to the current camera's field of view. Perform identity authentication within the field of view of each target camera. If the target employee appears within the field of view of any target camera and authentication is successful, designate the target camera as the new current camera, track the target employee, update the real-time model position of the target employee, and return to the blind spot prediction step. If the target employee does not appear within the field of view of any target camera or authentication fails, mark the personnel instance as an abnormal state in the digital twin model.

[0033] If no person appears within the shooting range of the next camera within a reasonable time frame, or if the person appears but does not match the target employee after identity verification (authentication failed), the system will execute the neighborhood expansion processing step.

[0034] In the neighborhood expansion processing step, all target cameras adjacent to the current camera's field of view are first identified. These target cameras are the adjacent nodes of the current camera in the park's camera topology. The logic for determining target cameras is similar to that of candidate cameras in the blind spot prediction step, and can be based on the topology map. After determining all target cameras, they are prioritized according to the spatial distance between each target camera and the current camera, as well as the park's traffic path constraints. For example, target cameras that are closer in space have higher priority; target cameras covering main roads or high-frequency traffic paths have higher priority than those covering remote paths; and cameras in directions where the target employee has frequently appeared in historical movement trajectories have higher priority than those in other directions. Finally, the system generates a search queue according to this priority and performs identity authentication within the shooting range of each target camera in turn.

[0035] The system retrieves video streams from each target camera sequentially according to the search queue. Within a reasonable timeframe, it detects personnel entering the frame and executes the same multimodal authentication process as in the blind spot crossing handling step. Once authentication is successful with a high-priority target camera, the search for subsequent target cameras can be terminated early to save computing resources. If the target employee is not within the shooting range of any target camera, or if a person appears in the surveillance footage but fails authentication, the system marks the person instance in the digital twin model as abnormal. Abnormal status can be achieved by changing the visual attributes of the person instance, such as changing the instance color from normal green to red, adding a flashing animation, or displaying an "Abnormal" text label above the instance. Simultaneously, the system can trigger an abnormality alarm, pushing alarm information to the park management terminal, including the target employee's identity information, the last known real-time model location, the current camera number, and the timestamp of the loss, so that security personnel can intervene and investigate promptly.

[0036] During the neighborhood expansion process, it's possible for multiple target cameras to simultaneously capture the target employee, and all of them may have successfully authenticated their identity. For example, the target employee might reach a road fork and be simultaneously captured by two target cameras covering two different branches, with both cameras successfully authenticating the employee. In this case, the system can select the optimal target camera from among the multiple target cameras as the new current camera. This selection can be based on factors such as the spatial distance between each target camera and the current camera, the confidence level of the target employee's pixel coordinates in each target camera's image, and the continuity of their historical movement trajectory. The system can then assign a weighted score to these factors and select the target camera with the highest score as the optimal camera. After selecting the optimal camera, this target camera is used as the new current camera to track the target employee and update the real-time model position. The system then returns to the blind spot prediction step and re-enters the main loop for tracking.

[0037] Further as Figure 1 As shown, the method may also include S60, namely the trajectory organization step: listen to the second card swipe event of the speed gate at the park exit, and according to the trigger time and card ID of the second card swipe event, obtain all real-time model positions of the target employee's personnel instance from the initial model position to the exit position corresponding to the second card swipe event; record all real-time model positions according to the time sequence to generate the target employee's movement trajectory in the park; and associate and store the movement trajectory with the target employee's identity information.

[0038] Specifically, when a target employee swipes their card at the exit speed gate, a second card-swiping event is generated. The system captures the trigger time and card ID of this event. Based on the trigger time and card ID of the second card-swiping event, the system retrieves all real-time model positions of the target employee's personnel instance corresponding to that card ID from the trajectory cache database, starting from the initial model position to the exit position corresponding to the second card-swiping event. These real-time model positions are model coordinate sequences continuously recorded and cached in the aforementioned collaborative tracking, blind spot crossing, and neighborhood expansion steps, with each coordinate point associated with a corresponding timestamp. The system then processes all real-time model positions according to the time sequence. The system generates the movement trajectory of the target employee within the park by connecting the rows. This trajectory can be represented as a sequence of model coordinate points ordered by time, or it can be fitted into a continuous three-dimensional spatial trajectory curve using a curve fitting algorithm. Finally, the movement trajectory is associated with the target employee's identity information and stored in the trajectory database, which includes at least the employee ID, name, entry time, exit time, trajectory point sequence (each point includes a timestamp, model coordinates X / Y / Z, and source camera number), total trajectory length, average movement speed, and the distribution of dwell time in each area. This information is used for subsequent management functions such as attendance verification, behavior analysis, and security auditing.

[0039] In actual use, multiple employees may enter or move within the park at the same time. The system will generate independent instances, determine the current camera, and track each employee independently, and update the location independently in real time. However, for the shared use of digital twin models, camera topology relationships, and video streams from various cameras, in order to avoid identity confusion, each personnel instance is bound to a unique internal tracking identifier, which corresponds one-to-one with the target employee's card ID and identity information.

[0040] Furthermore, in practical implementation, digital twin models can not only render personnel instances, but also map real-time footage from each camera as dynamic textures onto the surface of the corresponding camera model, or display video footage as a semi-transparent floating window above the ground in the monitored area, achieving a seamless integration of the video stream and the 3D model. When managers click on a camera model in the digital twin platform, the real-time footage from that camera pops up, overlaying the detection box and identity information of the currently tracked employee, further enhancing the visual monitoring experience.

[0041] This embodiment uses the listening of speed gate card swipe events as the trigger condition for real-time employee tracking. Combined with the corresponding employee identity and real-time location updates based on cameras, it can complete the real-time rendering of employee locations in the digital twin model without the need for additional equipment deployment, improving the overall management effect of employees in the park. Furthermore, it predicts the next camera by using camera topology relationships, avoiding the waste of computation caused by blindly searching the entire area. Finally, combined with a reasonable time range, it authenticates the identity of people entering within the window, realizing blind spot crossing, solving the interruption of target employee tracking under non-continuous coverage, and achieving deep integration of video and the digital twin model.

[0042] Based on the same inventive concept, the second embodiment of this disclosure provides a system for fusing video footage and 3D modeling based on a digital twin scene, applicable to a smart park with video surveillance. Its structural schematic diagram is shown below. Figure 2 As shown, it mainly includes: The instance generation module 10 is used to listen to the first card swipe event at the speed gate at the park entrance, determine the identity information and initial model position of the target employee based on the trigger time and card ID of the first card swipe event, create a personnel instance corresponding to the target employee, and render the personnel instance in real time at the initial model position in the digital twin model of the park, and set the camera that captures the initial model position as the current camera. The blind spot prediction module 20 is used to determine the next camera of the current camera based on the topology of the cameras in the park, and to determine whether there is a blind spot between the next camera and the current camera. The collaborative tracking module 30 is used to drive the current camera and the next camera to track the target employee when there is no blind spot between the next camera and the current camera, and update the real-time model position of the target employee. Based on the real-time model position, the rendering of the person instance in the digital twin model is updated, and the next camera is used as the new current camera. The blind spot crossing processing module 40 is used to drive the current camera to track the target employee and update the real-time model position of the target employee when there is a blind spot between the next camera and the current camera. It calculates a reasonable time range based on the time it takes for the target employee to disappear from the current camera's image and performs identity authentication on the person entering the next camera within the reasonable time range. When the authentication is successful, it drives the next camera to track the target employee and update the real-time model position of the target employee. It updates the rendering of the person instance in the digital twin model based on the real-time model position and uses the next camera as the new current camera. The neighborhood expansion processing module 50 is used to identify all target cameras adjacent to the current camera's field of view if no personnel enter the next camera within a reasonable time range or if authentication fails. Identity authentication is performed within the field of view of each target camera. If the target employee appears within the field of view of any target camera and authentication is successful, the target camera is designated as the new current camera. Target tracking is performed on the target employee, and the real-time model position of the target employee is updated. The process then returns to the blind spot prediction step. If the target employee does not appear within the field of view of any target camera or authentication fails, the personnel instance is marked as an abnormal state in the digital twin model.

[0043] In some embodiments, the system further includes an initialization module for establishing a digital twin model of the park, the digital twin model including three-dimensional models of park buildings, roads and access routes; marking the spatial position, shooting range and orientation angle of each camera in the digital twin model to establish the topological relationship of the park cameras; calibrating each camera to establish the mapping and transformation relationship between the pixel coordinate system of each camera and the model coordinate system of the digital twin model.

[0044] In some embodiments, the system further includes a trajectory processing module, which is used to monitor the second card swipe event at the park exit speed gate, and according to the trigger time and card ID of the second card swipe event, obtain all real-time model positions of the target employee's personnel instance from the initial model position to the exit position corresponding to the second card swipe event; record all real-time model positions according to the time sequence to generate the target employee's movement trajectory within the park; and associate and store the movement trajectory with the target employee's identity information.

[0045] It should be noted that the specific functions and principles implemented by each of the above modules have been explained in detail in the first embodiment, and will not be repeated here.

[0046] This embodiment uses the listening of speed gate card swipe events as the trigger condition for real-time employee tracking. Combined with the corresponding employee identity and real-time location updates based on cameras, it can complete the real-time rendering of employee locations in the digital twin model without the need for additional equipment deployment, improving the overall management effect of employees in the park. Furthermore, it predicts the next camera by using camera topology relationships, avoiding the waste of computation caused by blindly searching the entire area. Finally, combined with a reasonable time range, it authenticates the identity of people entering within the window, realizing blind spot crossing, solving the interruption of target employee tracking under non-continuous coverage, and achieving deep integration of video and the digital twin model.

[0047] Based on the same inventive concept, the second embodiment of this disclosure provides a storage medium storing a computer program, which, when executed by a processor, implements the steps of the method for fusing video footage and 3D modeling based on a digital twin scene as described in the first embodiment of this disclosure.

[0048] Based on the same inventive concept, the second embodiment of this disclosure provides an electronic device, including at least a memory and a processor. The memory stores a computer program, and when the processor executes the computer program in the memory, it implements the steps of the method for fusing video images and 3D modeling based on digital twin scenes as described in the first embodiment of this disclosure.

[0049] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this disclosure, and are not intended to limit them. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this disclosure.

Claims

1. A method for fusing video footage and 3D modeling based on a digital twin scene, characterized in that, Applications include smart parks with video surveillance, including: Instance generation steps: Listen for the first card swipe event at the speed gate at the park entrance, determine the identity information and initial model position of the target employee based on the trigger time and card ID of the first card swipe event, create a personnel instance corresponding to the target employee, and render the personnel instance in real time at the initial model position in the digital twin model of the park, and set the camera that captures the initial model position as the current camera. Blind spot prediction steps: Based on the topology of the cameras in the park, determine the next camera of the current camera, and determine whether there is a blind spot between the next camera and the current camera; Collaborative tracking steps: If there is no blind spot between the next camera and the current camera, drive the current camera and the next camera to track the target employee, update the real-time model position of the target employee, update the rendering of the person instance in the digital twin model according to the real-time model position, set the next camera as the new current camera and return to the blind spot prediction step. Blind Spot Crossing Processing Steps: In the event of a blind spot between the next camera and the current camera, the current camera is driven to track the target employee and update the real-time model position of the target employee. A reasonable time range is calculated based on the time it takes for the target employee to disappear from the current camera's view. Within the reasonable time range, the identity of the person entering the next camera is verified. If the verification is successful, the next camera is driven to track the target employee and update the real-time model position of the target employee. The rendering of the person instance in the digital twin model is updated based on the real-time model position. The next camera is then designated as the new current camera, and the process returns to the blind spot prediction step. Neighborhood expansion processing steps: If no personnel enter the next camera within the reasonable time range or authentication fails, identify all target cameras adjacent to the current camera's field of view. Perform identity authentication within the field of view of each target camera. If the target employee appears within the field of view of any target camera and authentication passes, designate that target camera as the new current camera, track the target employee, update the real-time model position of the target employee, and return to the blind spot prediction step. If the target employee does not appear within the field of view of any target camera or authentication fails, mark the personnel instance as an abnormal state in the digital twin model.

2. The fusion method according to claim 1, characterized in that, Prior to the instance generation step, the following is also included: Initialization steps: Establish a digital twin model of the park, which includes three-dimensional models of the park's buildings, roads, and access routes; In the digital twin model, the spatial location, shooting range, and orientation angle of each camera are marked to establish the topological relationship of the cameras in the park; Each camera is calibrated to establish a mapping and transformation relationship between the pixel coordinate system of each camera and the model coordinate system of the digital twin model.

3. The fusion method according to claim 2, characterized in that, The step of determining the next camera based on the campus camera topology includes: Based on the topology of the campus cameras, at least one candidate camera that is spatially adjacent to the current camera is identified; The historical movement trajectory of the target employee is determined based on the identity information; The next camera is determined from all the candidate cameras based on the historical movement trajectory and the travel path in the digital twin model.

4. The fusion method according to claim 3, characterized in that, The step of calculating a reasonable time range based on the time it takes for the target employee to disappear from the current camera's view includes: The average walking speed of the target employee is calculated based on the target employee's historical movement trajectory. Based on the digital twin model and the topological relationship of the campus cameras, the spatial distance between the current camera and the next camera is determined; Based on the average walking speed and the spatial distance, predict the walking time range of the target employee when passing through the blind spot; The reasonable time range is obtained by adding the walking time range to the disappearance time.

5. The fusion method according to claim 2, characterized in that, The step of updating the rendering of the person instance in the digital twin model according to the real-time model position includes: Extract the current pixel coordinates of the target employee from the real-time video stream captured by the current camera; Based on the mapping and transformation relationship between the pixel coordinate system and the model coordinate system, the current pixel coordinates are converted into model coordinates in the digital twin model, and the model coordinates are the real-time model position; The rendering position of the person instance is updated to the model coordinates, and the position of the person instance is smoothly interpolated and moved in the digital twin model according to the preset rendering frame rate.

6. The fusion method according to claim 2, characterized in that, The neighborhood expansion processing steps specifically include: After identifying all target cameras adjacent to the field of view of the current camera, the target cameras are prioritized according to the spatial distance between each target camera and the current camera and the park access path constraints. According to the priority order, identity authentication is performed sequentially within the shooting range of each target camera; When the target employee appears simultaneously within the shooting range of multiple target cameras and all identity authentications are successful, the optimal one is selected from the multiple target cameras as the new current camera based on the spatial distance between each target camera and the current camera, the confidence level of the pixel coordinates of the target employee in the shooting images of each target camera, and the trajectory continuity of the historical movement trajectory. The target employee is then tracked and the real-time model position of the target employee is updated. The process then returns to the blind spot prediction step. If the target employee is not within the shooting range of any of the target cameras or fails identity verification, the employee instance is marked as an abnormal state in the digital twin model.

7. The fusion method according to any one of claims 1 to 6, characterized in that, Also includes: Trajectory processing steps: Listen for the second card swipe event at the park exit speed gate. Based on the trigger time and card ID of the second card swipe event, obtain all real-time model positions of the target employee instance corresponding to the card ID from the initial model position to the exit position corresponding to the second card swipe event. Record the locations of all the real-time models according to the time series to generate the movement trajectory of the target employee within the park; The movement trajectory is associated with and stored in relation to the identity information of the target employee.

8. A system for fusing video footage and 3D modeling based on a digital twin scene, characterized in that, Applications include smart parks with video surveillance, including: The instance generation module is used to listen to the first card swipe event at the speed gate at the park entrance, determine the identity information and initial model position of the target employee based on the trigger time and card ID of the first card swipe event, create a personnel instance corresponding to the target employee, and render the personnel instance in real time at the initial model position in the digital twin model of the park, and set the camera that captures the initial model position as the current camera. The blind spot prediction module is used to determine the next camera of the current camera based on the topology of the cameras in the park, and to determine whether there is a blind spot between the next camera and the current camera. The collaborative tracking module is used to drive the current camera and the next camera to track the target employee when there is no blind spot between the next camera and the current camera, update the real-time model position of the target employee, update the rendering of the person instance in the digital twin model according to the real-time model position, and use the next camera as the new current camera. The blind spot crossing processing module is used to drive the current camera to track the target employee and update the real-time model position of the target employee when there is a blind spot between the next camera and the current camera. It calculates a reasonable time range based on the time it takes for the target employee to disappear from the current camera's view, and performs identity authentication on the person entering the next camera within the reasonable time range. When the authentication is successful, it drives the next camera to track the target employee and update the real-time model position of the target employee. Based on the real-time model position, it updates the rendering of the person instance in the digital twin model and sets the next camera as the new current camera. The neighborhood expansion processing module is used to determine all target cameras adjacent to the current camera's field of view if no personnel enter the next camera within the reasonable time range or if authentication fails. Identity authentication is performed within the field of view of each target camera. If the target employee appears within the field of view of any target camera and authentication is successful, that target camera is designated as the new current camera. Target tracking is performed on the target employee, and the real-time model position of the target employee is updated. The process then returns to the blind spot prediction step. If the target employee does not appear within the field of view of any target camera or authentication fails, the personnel instance is marked as abnormal in the digital twin model.

9. A storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the method for fusing video footage and 3D modeling based on a digital twin scene as described in any one of claims 1 to 7.

10. An electronic device, comprising at least a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program on the memory, it implements the steps of the method for fusing video footage and 3D modeling based on a digital twin scene as described in any one of claims 1 to 7.

Images