Digital twin positioning method and system based on BIM and video

Abstract

The present invention discloses a digital twin positioning method and system based on BIM and video, including: the present invention first collects a real-time first video stream of a construction site, adjusts the frequency of extracting frame images according to the structural similarity index SSIM between frames to form a second video stream; uses a trained convolutional neural network CNN to extract features from the second video stream, and inputs it into a recurrent neural network RNN, obtains the dynamic change features of the frame sequence through the recurrent neural network RNN, identifies people and objects according to the dynamic change features of the frame sequence and matches them with the element data in the BIM model, and then updates the digital twin model in real time to reflect the real-time status of the construction site. The present application adjusts the video frame extraction frequency of the current time window according to the average value of the SSIM value in the time window, and dynamically adjusts the activation function according to the video frame extraction frequency during the feature extraction process, which greatly improves the accuracy and efficiency of BIM model generation.

Description

Technical Field

[0001] The present invention relates to the technical field of BIM positioning, and in particular to digital twin positioning based on BIM and video. Background Art

[0002] In the field of modern building management and security monitoring, building information modeling technology and video surveillance systems provide strong support for the design, construction, operation and maintenance of buildings. Digital twin technology simulates dynamic processes in the real world in virtual space by creating virtual models of physical objects. This technology was originally used in the manufacturing and aerospace fields, and has gradually expanded to the construction industry in recent years. In the construction industry, digital twins not only reflect the structure and function of the building, but also can be updated in real time to reflect the real-time status of the building and provide decision support for building life cycle management. BIM is a revolutionary technology in the construction industry. It goes beyond traditional two-dimensional design drawings and provides a real three-dimensional digital model, including the building's geometric structure, spatial relationships, geographic information, and attribute information related to the building life cycle. The use of BIM makes project management more efficient because it provides detailed information to support building design, construction, and operation decisions.

[0003] Video surveillance systems are an important part of modern building security systems. They can provide real-time data to monitor the security status inside and outside the building. Video streams captured by high-definition cameras are a key data source for monitoring building security. Traditionally, video surveillance data is mainly used for post-analysis and real-time security monitoring, but the potential of its data is far from being fully utilized. Although BIM and video surveillance systems are effective in their own way, the lack of effective data and view integration between them limits the operator's comprehensive understanding of the building status. For example, video surveillance captures an emergency, but the operator cannot immediately determine the exact location of the incident and related building information. Therefore, integrating video data with BIM models in real time to create a digital twin model that can reflect the real-time status of the building is of great value in improving building operation efficiency and optimizing safety response mechanisms.

[0004] Although the existing technology integrates video data and BIM models, the amount of video stream image data is huge. When combined with the BIM model, the video data stream is not extracted and simplified, resulting in poor processing efficiency and effect. In addition, the video stream extraction process in the existing technology does not take into account the degree of change of the video content, resulting in the extraction frequency The extracted video frames are not representative, resulting in poor BIM model mapping results. In the process of extracting the extracted video stream features, the influence of the extraction frequency on the video stream is not considered, resulting in a greatly reduced accuracy of BIM model position generation. In view of the limitations of the existing BIM model mapping, a new solution is urgently needed for high-accuracy and high-efficiency video stream extraction and automated processing methods to improve processing efficiency and customer satisfaction. Summary of the invention

[0005] In view of the above problems mentioned in the prior art, the present invention provides a digital twin positioning method and system based on BIM and video. The method firstly collects a real-time first video stream of the construction site with a camera and performs preprocessing, performs frame serialization on the preprocessed real-time first video stream, and adjusts the frequency of extracting frame images according to the structural similarity index SSIM between frames to form a second video stream; uses a trained convolutional neural network CNN to extract features of the second video stream, extracts the features of the second video stream, inputs the extracted features of the second video stream into a recursive neural network RNN, obtains the dynamic change features of the frame sequence through the recursive neural network RNN, uses a softmax function to identify and classify the dynamic change features of the frame sequence output by the recursive neural network RNN, obtains the people and objects in the video frame, matches the identified people and objects with the element data in the BIM model, and updates the digital twin model in real time according to the matching results to reflect the real-time status of the construction site. This application uses the average value I of the SSIM value in the time window to obtain the dynamic change features of the frame sequence output by the recursive neural network RNN, and obtains the dynamic change features of the frame sequence output by the recursive neural network RNN. avg The video frame extraction frequency f of the current time window is adjusted. During the feature extraction process, the improved ModReLU activation function is dynamically adjusted according to the video frame extraction frequency, which greatly improves the accuracy and efficiency of BIM model generation and greatly enhances the user experience.

[0006] This application provides a digital twin positioning method based on BIM and video, including the steps of:

[0007] S1: The camera collects a real-time first video stream of the construction site and pre-processes the first video stream data;

[0008] S2: using a recursive neural network (RNN) to process the collected first video stream to identify people and objects in the video;

[0009] S21: performing frame serialization on the preprocessed real-time first video stream, decomposing the first video stream into a series of separate frames, and adjusting the frequency of extracting frame images according to a structural similarity index SSIM between frames;

[0010] S211: Calculate the structural similarity index SSIM between consecutive frame pairs within the set time window, and find the average value I of the SSIM value within the set time window avg ;

[0011] S212: According to the average value of SSIM value in the time window I avg Adjust the video frame extraction frequency f of the current time window,

[0012] Among them, T maxis the longest interval of video frame extraction; T min The shortest interval time for video frame extraction;

[0013] S213: splicing the extracted video frames to form a second video stream;

[0014] S22: using the trained convolutional neural network CNN to perform feature extraction on the second video stream, thereby extracting features of the second video stream;

[0015] S23: inputting the extracted second video stream features into a recurrent neural network RNN, and obtaining dynamic change features of the frame sequence through the recurrent neural network RNN;

[0016] S24: Using the softmax function to identify and classify the dynamic change features of the frame sequence output by the recurrent neural network RNN, and obtain the characters and objects in the video frame;

[0017] S3: matching the identified people and objects with element data in the BIM model, including using a point position matching algorithm to match the positions of people and objects identified in the video stream with rooms or floors in the BIM model;

[0018] S4: Based on the matching results, the digital twin model is updated in real time to reflect the real-time status of the construction site;

[0019] S5: Display the updated digital twin model on the display device.

[0020] Preferably, in S1, the camera collects real-time video stream of the construction site and pre-processes the video stream data, including denoising the video stream by using Gaussian filtering.

[0021] Preferably, S22: using a trained convolutional neural network CNN to extract features from the second video stream, extracting features of the second video stream; the features of the second video stream include edges and colors of people and objects; the trained convolutional neural network CNN uses an improved ModReLU activation function, and the improved ModReLU activation function is dynamically adjusted according to the video frame extraction frequency,

[0022] ModReLU(x,f)=max(0,x-β0*e -k*f );

[0023] Among them, β0 is the initial adjustment parameter, k is the exponential adjustment parameter, e is the base of the natural logarithm, f is the video frame extraction frequency, and x is the output of the convolution layer.

[0024] Preferably, S23: inputting the extracted second video stream features into a recursive neural network RNN, obtaining the dynamic change features of the frame sequence through the recursive neural network RNN, including the recursive neural network calculating the current hidden state based on the second video stream features and the previous hidden state, thereby continuously updating the internal state to reflect the dynamic changes in the video frame sequence, the recursive neural network extracting the dynamic change features of the frame sequence from the hidden state of each time step, the dynamic change features of the frame sequence including motion trajectories, and the motion trajectories are marked with spatial coordinates.

[0025] Preferably, S24: using a softmax function to identify and classify the dynamic change features of the frame sequence output by the recursive neural network RNN ​​to obtain the people and objects in the video frame; applying the softmax function to process the dynamic change features of the frame sequence to calculate the probabilities of the people and objects, among which the one with the highest probability is the final classification result.

[0026] Preferably, S3: matching the identified people and objects with element data in the BIM model, including using a point position matching algorithm to correspond the positions of people and objects identified in the video stream to the rooms or floors in the BIM model; including converting the identified coordinates or bounding boxes into three-dimensional space coordinates of the BIM model based on the identified positions of people and objects, that is, the identified coordinates or bounding boxes of people and objects in the video frame; and then determining the exact room or floor positions of the people and objects identified in the video in the BIM model.

[0027] Preferably, S4: based on the matching results, updating the digital twin model in real time to reflect the real-time status of the construction site, including: modifying the position and status of elements corresponding to people and objects in the digital twin model, which elements reflect the latest status of people and objects identified in the video surveillance data.

[0028] Preferably, converting the identified coordinates or bounding box into three-dimensional space coordinates of the BIM model includes converting two-dimensional image coordinates into three-dimensional space coordinates by a back projection method according to calibration data of the camera, wherein the calibration data includes the position, orientation, focal length and lens distortion parameters of the camera.

[0029] The present application also provides a digital twin positioning system based on BIM and video, including:

[0030] The acquisition module uses a camera to acquire a real-time first video stream of the construction site and pre-process the first video stream data;

[0031] The recognition module uses a recursive neural network (RNN) to process the collected first video stream to recognize people and objects in the video;

[0032] A serialization module performs frame serialization on the preprocessed real-time first video stream, decomposes the first video stream into a series of separate frames, and adjusts the frequency of extracting frame images according to a structural similarity index SSIM between frames;

[0033] The similarity index calculation module calculates the structural similarity index SSIM between consecutive frame pairs within the set time window and calculates the average value of the SSIM value within the set time window. avg ;

[0034] The extraction frequency determination module is based on the average value of the SSIM value in the time window. avg Adjust the video frame extraction frequency f of the current time window,

[0035] Among them, T max is the longest interval of video frame extraction; T min The shortest interval time for video frame extraction;

[0036] A second video stream forming module, splicing the extracted video frames to form a second video stream;

[0037] A second video stream feature extraction module uses a trained convolutional neural network CNN to perform feature extraction on the second video stream to extract features of the second video stream;

[0038] The frame sequence dynamic change feature extraction module inputs the extracted second video stream features into the recursive neural network RNN, and obtains the frame sequence dynamic change features through the recursive neural network RNN;

[0039] The recognition and classification module uses the softmax function to recognize and classify the dynamic change features of the frame sequence output by the recurrent neural network RNN ​​to obtain the people and objects in the video frame;

[0040] A position determination module matches the identified people and objects with element data in the BIM model, including using a point position matching algorithm to match the positions of people and objects identified in the video stream with rooms or floors in the BIM model;

[0041] The update module updates the digital twin model in real time based on the matching results to reflect the real-time status of the construction site;

[0042] The display module displays the updated digital twin model on the display device.

[0043] Preferably, the second video stream feature extraction module uses a trained convolutional neural network CNN to extract features from the second video stream to extract features of the second video stream; the second video stream features include edges and colors of people and objects; the trained convolutional neural network CNN uses an improved ModReLU activation function, and the improved ModReLU activation function is dynamically adjusted according to the video frame extraction frequency.

[0044] ModReLU(x,f)=max(0,x-β0*e -k*f );

[0045] Among them, β0 is the initial adjustment parameter, k is the exponential adjustment parameter, e is the base of the natural logarithm, f is the video frame extraction frequency, and x is the output of the convolution layer.

[0046] The present invention provides a digital twin positioning method and system based on BIM and video, which can achieve the following beneficial technical effects:

[0047] 1. The present invention first collects a real-time first video stream of a construction site with a camera and performs preprocessing, performs frame serialization on the preprocessed real-time first video stream, adjusts the frequency of extracting frame images according to the structural similarity index SSIM between frames to form a second video stream; uses a trained convolutional neural network CNN to extract features of the second video stream, extracts features of the second video stream, inputs the extracted features of the second video stream into a recursive neural network RNN, obtains dynamic change features of the frame sequence through the recursive neural network RNN, uses a softmax function to identify and classify the dynamic change features of the frame sequence output by the recursive neural network RNN, obtains people and objects in the video frame, matches the identified people and objects with the element data in the BIM model, and updates the digital twin model in real time according to the matching results to reflect the real-time status of the construction site. This application uses the average value of the SSIM value within the time window I avg The video frame extraction frequency f of the current time window is adjusted. During the feature extraction process, the improved ModReLU activation function is dynamically adjusted according to the video frame extraction frequency, which greatly improves the accuracy and efficiency of BIM model generation and greatly enhances the user experience.

[0048] 2. The present invention performs frame serialization on the preprocessed real-time first video stream, decomposes the first video stream into a series of separate frames, and adjusts the frequency of extracting frame images according to the structural similarity index SSIM between frames; calculates the structural similarity index SSIM between consecutive frame pairs within a set time window, and calculates the average value I of the SSIM value within the set time window. avg ; According to the average value of SSIM value in the time window I avg Adjust the video frame extraction frequency f of the current time window,

[0049] Among them, T max is the longest interval of video frame extraction; T min The shortest interval time of video frame extraction; This application calculates the structural similarity index SSIM between consecutive frame pairs, and calculates the average value of SSIM value in the set time window I avg Calculate the video frame extraction frequency. When the difference is large, it means that there are changes in people or objects. Increasing the extraction frequency greatly improves the accuracy of BIM model generation.

[0050] 3. The convolutional neural network CNN trained by the present invention adopts an improved ModReLU activation function, and the improved ModReLU activation function is dynamically adjusted according to the video frame extraction frequency.

[0051] ModReLU(x,f)=max(0,x-β0*e -k*f );

[0052] Among them, β0 is the initial adjustment parameter, k is the exponential adjustment parameter, e is the base of the natural logarithm, f is the video frame extraction frequency, and x is the output of the convolution layer; the present invention combines the video frame extraction frequency and adds it into the activation function calculation process. The improved activation function enables CNN to adjust its processing characteristics according to the dynamic changes of the video frame, enhances the flexibility and efficiency of processing video streams, and improves the adaptability and accuracy of the system to rapidly changing scenes. The video frame extraction frequency is integrated into the activation function of CNN as a dynamic parameter, so that the network can dynamically adjust its activation threshold according to the changes between frames when processing video frames, thereby more flexibly responding to changes in video content, greatly improving the accuracy and efficiency of BIM model position prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0054] Figure 1 It is a schematic diagram of the steps of the digital twin positioning method based on BIM and video of the present invention;

[0055] Figure 2 It is a schematic diagram of the digital twin positioning system based on BIM and video of the present invention. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] Embodiment 1:

[0058] In view of the above problems mentioned in the prior art, in order to solve the above technical problems, as shown in the attached Figure 1 As shown: This application provides a digital twin positioning method based on BIM and video, comprising the steps of:

[0059] S1: The camera collects the real-time first video stream of the building site and pre-processes the first video stream data; in some embodiments, the camera is deployed at a key position to capture the video of the building site in real time. These cameras are usually configured at the entrance, exit, main passage and other key areas to comprehensively monitor the safety and operation status of the building; in order to ensure sufficient image quality and real-time performance, the camera is usually set to a high resolution (such as 1080p or higher) and an appropriate frame rate (such as 30 frames per second), which makes the video sufficient to show the movements of people and objects in detail; digital filtering technology, such as Gaussian filtering, is used to remove random noise in the video data. This step helps to improve the accuracy of subsequent image processing and target detection steps; adjust brightness, contrast and saturation to adapt to different ambient light conditions to ensure that the video image is clearly visible under different lighting conditions; correct video jitter caused by camera vibration caused by wind or other external factors; convert the video stream into a format and encoding suitable for processing and analysis, and convert it from the original H.264 to a frame sequence to facilitate frame-by-frame analysis in subsequent steps. In some embodiments, a full HD 1080p camera is installed at the building entrance, and the camera is set to capture video at a rate of 30 frames per second; the camera is connected to a central monitoring system and transmits video data in real time. The pre-processed video uses a Gaussian filtering algorithm to remove random noise in the image caused by environmental factors (such as rain and fog); the brightness and contrast of the video are automatically adjusted according to the current ambient light conditions to ensure that the video has good visibility at different time periods; video stabilization technology is applied to reduce image jitter and improve video quality; the video data is converted into a format suitable for further processing, and each frame of the image is ready for subsequent feature extraction and analysis.

[0060] S2: Using a recursive neural network RNN ​​to process the first video stream collected to identify people and objects in the video; in some embodiments, the video frames of the RNN are first preprocessed in video recognition, including resizing and normalization, and then input into one or more convolutional layers to extract features of each frame. These features include visual information such as edges, corners, textures, etc., which provide input for subsequent RNN processing; the extracted feature sequence is then fed into the RNN. The RNN saves the information of the previous frame through its internal state (hidden layer), combines this information with the input of the current frame, so that the information in the time series can be comprehensively analyzed; at each time step, the RNN updates its internal state and can output the processing result of the current frame, such as the identified person or object; this processing method based on the information of the previous and next frames enables the RNN to more effectively identify dynamic objects in the video. In some embodiments, step 1: high-resolution cameras are installed at each key entrance and exit to continuously collect video streams. Step 2: The video stream is transmitted to the central processing unit in real time for denoising and normalization. Step 3: Use a convolutional neural network (CNN) to extract key visual features from each frame of video. Step 4: Input the features of consecutive frames into the RNN. The RNN structure is based on LSTM (Long Short-Term Memory) or GRU (Gated Recurrent Unit) to optimize the processing of time-dependent information in the video. The RNN evaluates the dynamics of the person in each frame, such as walking, running or other actions, and identifies specific objects, such as bags or other items carried. Step 5: The recognition results are updated in real time to the digital twin model, allowing the security team to see each identified individual and their real-time location in the shopping mall on the display screen in the control room.

[0061] S21: Frame serialization is performed on the preprocessed real-time first video stream, the first video stream is decomposed into a series of separate frames, and the frequency of extracting frame images is adjusted according to the structural similarity index SSIM between the frames; in some embodiments, frames are first extracted from the real-time video stream, and each video frame is usually extracted at a fixed time interval (such as 30 frames per second); in some embodiments, the SSIM of two adjacent frames is calculated, two consecutive frames are selected, the SSIM value between them is calculated, and the degree of change of the content of the two frames is determined. If the SSIM value is high (close to 1), it indicates that the change between the two frames is small, and the frame extraction frequency can be reduced; if the SSIM value is low, it indicates that the change between the two frames is large, and the frame extraction frequency needs to be increased to capture more details; the frame extraction frequency is adjusted, and the extraction frequency of subsequent frames is dynamically adjusted according to the SSIM calculation result, and a threshold is set. When the SSIM value is greater than 0.95, the extraction frequency is reduced; when the SSIM value is less than 0.8, the extraction frequency is increased.

[0062] S211: Calculate the structural similarity index SSIM between consecutive frame pairs within the set time window, and find the average value I of the SSIM value within the set time windowavg ; In some embodiments, the SSIM index is an indicator for measuring the similarity between two images, which compares three aspects of the two images: brightness, contrast, and structure; the brightness comparison calculates the average brightness of the two images; the contrast comparison calculates the standard deviation of the contrast of the two images; and the structure comparison calculates the covariance of the two images multiplied by their respective standard deviations.

[0063]

[0064] u x 、u y is the average value of images x and y, σ x , σ y is the standard deviation of images x and y, σ xy is the covariance of images x and y, C1 and C2 are the first and second small constants added to avoid the denominator being zero.

[0065] In some embodiments, step 1: define a time window, such as every minute or every five minutes, and the system will calculate the SSIM values ​​of all consecutive frame pairs during this period. Step 2: calculate the SSIM values ​​between consecutive frame pairs, collect video frames in each time window, such as 30 frames per second, and for each pair of consecutive frames, calculate the SSIM value between them, the first frame and the second frame, the second frame and the third frame, and so on. Step 3: calculate the average of the SSIM value, add up all the calculated SSIM values, and then divide them by the total number of frame pairs to obtain the average value of the SSIM in the time window. If the average SSIM value is high (for example, greater than 0.95), it indicates that the video content does not change much, and the frame rate can be reduced to save resources. If the average SSIM value is low (for example, lower than 0.85), it indicates that the video content changes greatly, and the frame rate should be increased to capture more details. Step 4: dynamically adjust the frame extraction frequency, and dynamically adjust the frame extraction frequency of the next time window according to the SSIM average value. If the SSIM average value is high, reduce the frame rate; if the SSIM average value is low, increase the frame rate.

[0066] S212: According to the average value of SSIM value in the time window I avg Adjust the video frame extraction frequency f of the current time window,

[0067] Among them, T max is the longest interval of video frame extraction; T minThe shortest interval time for video frame extraction; the system will dynamically adjust the frequency of video frame extraction based on the previously calculated SSIM average value. Simply put, if the video content changes little (i.e., the SSIM value is high), it means that the scene is relatively static, and the frame extraction frequency can be reduced to save storage and processing resources; conversely, if the video content changes greatly (i.e., the SSIM value is low), the frame extraction frequency needs to be increased to capture important dynamic information. In one embodiment, before the start of the event, the system captures video frames at the highest frequency (e.g., 30 frames per second) to monitor the entry of spectators; as the spectators gradually take their seats, the activities in the venue decrease, and the changes in the content of the monitored video begin to decrease. The system calculates the average SSIM value between all consecutive frame pairs every one minute; if the calculated SSIM average value shows that the scene is relatively stable (such as the SSIM average value is higher than 0.95), the system will automatically reduce the frame extraction frequency, for example, from 30 frames per second to 10 frames per second; when the game is in progress or in an emergency, the audience's activity increases, the SSIM average value will decrease (for example, below 0.8), and the system will increase the frame extraction frequency accordingly and readjust it to 30 frames per second to ensure that all important events are captured and recorded; in this way, the system can optimize the use of storage resources and processing bandwidth while ensuring the quality of monitoring, which is very effective in long-term monitoring.

[0068] S213: splicing the extracted video frames to form a second video stream; the frames extracted from the original video stream are recombined to form a new, more efficient video stream, which includes all key events and changes, while ignoring parts with very small content changes; the system recombines the selected frames (i.e., those extracted during the dynamic frame rate adjustment process) to form a second video stream; the video stream now contains a record of key changes and events, but does not include static or repeated content, thereby significantly reducing the amount of data and the computing resources required for subsequent processing.

[0069] S22: Use the trained convolutional neural network CNN to extract features from the second video stream, and extract the features of the second video stream; Convolutional neural network (CNN) is an efficient image and video analysis tool in deep learning, and is particularly suitable for automatically extracting complex features from visual data; In the application of video analysis, CNN can identify and abstract important features in the image, such as shape, edge, texture, etc.; The features extracted from the video include but are not limited to the shape, size, color and motion information of the object; These features are essential for understanding the content of the video, tracking the target and analyzing the behavior; CNN automatically learns useful feature representations from the original pixels through its multi-layer structure; The first layer only captures simple edges and corners, while deeper layers can recognize more complex object features such as facial parts or entire objects. In some embodiments, the following steps are included: Step 1 Through the previous step S21, a second video stream has been generated from the original video stream according to the dynamic changes of the content. This stream focuses on key activities and time periods with large changes, thereby reducing the amount of data; Step 2 Input the second video stream into a trained CNN model, which is trained to recognize specific types of behaviors in the video, such as running, cluster formation or abnormal behavior. CNN processes video frames through its multiple convolutional layers to extract features that are highly descriptive and helpful for subsequent analysis. The features extracted in step 3 are used for further analysis, such as using machine learning models to identify specific behavior patterns or potential security threats. The identification results can be used to automatically trigger alarm systems or notify security personnel. In step 4, the results of identification and analysis are updated to the digital twin model of the mall, reflecting the mall’s security status in real time and providing a visual interface for the security team to monitor and respond.

[0070] S23: Input the extracted second video stream features into the recurrent neural network RNN, and obtain the dynamic change features of the frame sequence through the recurrent neural network RNN; RNN is particularly suitable for processing data that is continuous in time, which makes it very suitable for video analysis tasks, in which each video frame is not isolated, but associated with the previous and next frames. RNN can remember the output of the previous state and process it together with the current input, which is crucial for understanding the behavior patterns and motion trajectories in the video. In some embodiments, step 1 feature extraction is installed in the key position of the airport. The camera captures the video stream, and after the first round of processing, the second video stream is generated. This stream has targetedly included more video clips of potential important events. CNN is used to extract key visual features from these video clips, such as details such as character shape and luggage items. Step 2 Input the features extracted by CNN into RNN. When processing these data, RNN can analyze and identify patterns in the time series, such as individual motion paths, group dynamics, etc. RNN analyzes features frame by frame, memorizes and updates its internal state, thereby achieving continuous tracking of dynamic changes. Step 3 The output of RNN is used to identify specific behavior patterns, such as fast movement indicating an emergency, or specific cluster behavior indicating potential safety risks. Through real-time analysis, the system can issue early warnings before critical events occur, such as sending notifications to security personnel and adjusting airport operation strategies. Step 4 All identified information and analysis results are updated in real time to the airport's digital twin model, which provides a comprehensive view of the airport's real-time security status.

[0071] S24: Use the softmax function to identify and classify the dynamic change features of the frame sequence output by the recursive neural network RNN ​​to obtain the characters and objects in the video frame; the softmax function is a commonly used function in machine learning classification problems. It can convert one or more real numbers into probability distributions. The output feature vector of each frame can be interpreted as the predicted probability of each category after being converted by the softmax function. In one embodiment, the following steps are included: Step 1 The video captured by the camera in the store is serialized and extracted by CNN. These features are processed by RNN to analyze the continuity and dynamic changes in time. Step 2 The output features after RNN processing include information about the trajectory of the character's movement, the movement of the object, etc. These dynamically changing features are fed into the softmax function. Step 3 The softmax function converts the feature vector of each frame into a set of probability distributions. For example, the output includes: customers (70%), employees (30%). Step 4 According to the category with the highest probability output by softmax, the system can automatically mark the characters in the video frame.

[0072] S3: Match the identified people and objects with the element data in the BIM model, including using a point position matching algorithm to match the positions of the people and objects identified in the video stream with the rooms or floors in the BIM model; in some embodiments, the point position matching algorithm is mainly used to convert the two-dimensional coordinates identified from the video into three-dimensional spatial coordinates in the BIM model, and find the specific positions of these coordinates in the building (such as specific room numbers or floors); through the image processing and machine learning techniques in the previous step, such as CNN and RNN, the system can identify people and objects in the video and determine their positions in the video frame. Usually, these position information is given in the form of two-dimensional pixel coordinates. Use the camera's calibration data (including information such as position, orientation, and focal length) to convert the two-dimensional image coordinates into three-dimensional world coordinates. This involves complex geometric and perspective transformation calculations; match the converted three-dimensional coordinates with the spatial data in the BIM model; this step requires the BIM model to be accurate and include sufficient metadata to support spatial queries, such as floor divisions, room numbers, etc. In some embodiments, step 1 video surveillance and character recognition, cameras are installed in key areas of the building, such as the lobby, emergency exits, etc. These cameras continuously monitor and transmit video to the central monitoring system in real time. The system analyzes the video stream in real time, identifies the characters in the picture, and determines their positions in the video frame. Step 2, use the camera calibration data to convert the two-dimensional position of each character into the three-dimensional spatial coordinates of the office building, match these three-dimensional coordinates with the BIM model, and determine the specific location of each character in the building, such as a specific room or floor. Step 3 When the system detects any unusual or emergency situation, such as unauthorized entry or sudden gathering of people, the system can immediately notify security personnel and highlight the relevant areas on the BIM model; security personnel can quickly understand the situation and respond through the detailed BIM model, such as guiding people to evacuate or go directly to the scene of the incident.

[0073] S4: Based on the matching results, the digital twin model is updated in real time to reflect the real-time status of the building site; the digital twin model is a dynamic, real-time virtual model that reflects the physical environment. It integrates various sensors and monitoring data, including data from video surveillance. When the monitoring system identifies a key event or state change, this information is used to update the digital twin model so that the model always reflects the current reality. In some embodiments, step 1, the video data captured by the camera is processed to identify the positions of people and objects in the image. These data are matched with the spatial data in the BIM model through the previous steps. Once the system detects a specific event, such as customers gathering during non-business hours or abnormal gathering near the emergency exit, the system immediately identifies this information as a key event. Step 2 The system updates the digital twin model based on the data obtained from the video analysis. For example, if an area is monitored to have an increased security risk (such as a fire alarm is triggered), the digital model marks this area as high risk and updates the status information in the model in real time. The updated information includes the location of the personnel, the direction of movement, the use of the emergency exit, etc. Step 3 The updated digital twin model is displayed to the security management team in real time through a graphical user interface. The management team can see the status of each area, as well as all relevant safety events and warnings. Based on the information provided by the model, the management team can make decisions quickly, such as adjusting evacuation routes, dispatching security personnel to specific areas, or providing guidance to customers through the broadcast system.

[0074] S5: Display the updated digital twin model on a display device. Visualization of a digital twin model is a process of converting complex data and models into an easy-to-understand graphical interface. It usually includes a three-dimensional model of the building, an overlay of real-time data (such as temperature, traffic flow, etc.), and any key warnings or indicators. Such visualization makes the model not only a collection of data, but also an intuitive and interactive decision support tool. In some embodiments, a large-screen display device displays the updated digital twin model in real time. The model not only displays real-time data, such as the current traffic flow and the degree of congestion in a specific area, but also displays any alarms triggered by the video surveillance system in real time.

[0075] In some embodiments, the S1: camera collects real-time video streams of the construction site and pre-processes the video stream data, including denoising the video stream using Gaussian filtering. Gaussian filtering is a smoothing technique based on the Gaussian function (normal distribution function), which smoothes the image by assigning different weights to the surrounding pixels (the center pixel has the highest weight, and the farther the pixel, the lower the weight). This weighted averaging method can effectively reduce random noise in the image, especially the noise generated under low light conditions or when the camera quality is poor. In some embodiments, the video data collected by each camera is first denoised by a Gaussian filter, using a 5x5 Gaussian kernel, calculating the weight according to the spatial distance between pixels, and performing weighted averaging on the surrounding pixels of each pixel, effectively removing random noise in the image. The video data processed by Gaussian filtering has fewer noise points and smoother image quality, which helps to improve the accuracy of subsequent video analysis.

[0076] In some embodiments, the S22: uses a trained convolutional neural network CNN to extract features from the second video stream to extract features of the second video stream; the features of the second video stream include edges and colors of people and objects; the trained convolutional neural network CNN uses an improved ModReLU activation function, and the improved ModReLU activation function is dynamically adjusted according to the video frame extraction frequency,

[0077] ModReLU(x,f)=max(0,x-β0*e -k*f );

[0078] Among them, β0 is the initial adjustment parameter, k is the exponential adjustment parameter, e is the base of the natural logarithm, f is the video frame extraction frequency, and x is the output of the convolutional layer. ModReLU is a variant of the ReLU activation function, which was originally designed for complex inputs, but in this application, its improved version is considered to be suitable for real inputs and can self-adjust according to the dynamics of the input. Specifically, the behavior of ModReLU can respond to changes in the video frame extraction frequency by adjusting its activation threshold, making the network more sensitive to important feature changes in rapidly changing scenes. The extracted video frames are input into the trained CNN, which uses the improved ModReLU activation function for feature extraction. The CNN is specially optimized for identifying and analyzing features such as edges and colors of vehicles and pedestrians, as well as changes in background and weather conditions; the ModReLU activation function dynamically adjusts its parameters according to the video frame extraction frequency to capture important information in rapidly changing traffic scenes.

[0079] In some embodiments, the S23: input the extracted second video stream features into the recurrent neural network RNN, and obtain the dynamic change features of the frame sequence through the recurrent neural network RNN, including the recurrent neural network calculating the current hidden state according to the second video stream features and the previous hidden state, thereby continuously updating the internal state to reflect the dynamic changes in the video frame sequence, and the recurrent neural network extracts the dynamic change features of the frame sequence from the hidden state of each time step, and the dynamic change features of the frame sequence include motion trajectories, and the motion trajectories are marked with spatial coordinates. Recurrent neural networks (RNNs) are used to process the optimized second video stream, aiming to capture and analyze dynamic change features from video data. Through the continuous time step processing of RNN, dynamic events in the video, such as the movement trajectory of people or objects, can be effectively tracked and understood. This analysis is particularly important for application scenarios that require real-time monitoring and response. Through RNN, important dynamic change features can be extracted from the hidden state of each time step, and the motion trajectory tracks the movement path of the object in the video. Speed ​​and direction changes analyze the changes in the movement speed and direction of the object. Behavior pattern recognition such as walking, running, jumping and other specific behavior patterns. The features extracted by CNN are input into RNN, which analyzes these features and their changes over time to identify key dynamic changes, such as the movement patterns and crowded areas of the crowd. RNN draws the movement trajectories of the crowd by considering the dependencies between previous and next frames in the time series and marks the spatial coordinates of these trajectories.

[0080] In some embodiments, the S24: using the softmax function to identify and classify the dynamic change features of the frame sequence output by the recursive neural network RNN, and obtaining the characters and objects in the video frame; applying the softmax function to the dynamic change features of the frame sequence, and calculating the probabilities of the characters and objects, wherein the highest probability is the final classification result. The dynamic change features of the frame sequence output by the recursive neural network (RNN) are processed by the softmax function to identify and classify the characters and objects in the video frame. The application of the softmax function is to convert the output of the RNN into a set of probability distributions, which represent the recognition probabilities of various types of objects (such as different types of characters and objects) in the video frame. By comparing these probability values, the system can determine the most likely classification result in each frame. The softmax function is a commonly used activation function in multi-class classification, which can "compress" a K-dimensional vector containing any real number into another K-dimensional real vector, where each element in the vector ranges from 0 to 1, and the sum of all elements is 1. Therefore, the output of the softmax function can be regarded as a probability distribution, which is suitable for probability prediction in multi-class classification problems. In one embodiment, the output of the RNN is the feature vectors of various types of luggage and behaviors in each frame. These vectors are converted into probability distributions through the softmax function. For a frame of image, the softmax output is: handbag (0.70), suitcase (0.10), backpack (0.20); based on the softmax output, the system determines that the most likely object in this frame is the handbag.

[0081] In some embodiments, the S3: matching the identified people and objects with the element data in the BIM model includes using a point position matching algorithm to correspond the positions of the people and objects identified in the video stream to the rooms or floors in the BIM model; including converting the identified coordinates or bounding boxes into three-dimensional space coordinates of the BIM model according to the positions of the identified people and objects, that is, the coordinates or bounding boxes of the people and objects in the identified video frames; and then determining the exact room or floor positions of the people and objects identified in the video in the BIM model. In some embodiments, the camera captures the video inside the hospital in real time, and uses a deep learning model (such as a convolutional neural network) to analyze these video streams to automatically identify the people (such as doctors, patients, visitors) and objects (such as medical equipment, wheelchairs) in the video frames. Once the objects are identified, the system extracts their coordinates or bounding boxes in the video frame. Using the calibration parameters of the camera (including position, orientation, focal length, etc.), these two-dimensional coordinates are converted into three-dimensional space coordinates of the hospital. The converted three-dimensional coordinates are matched with the BIM model. The system determines the exact location of each identified object by comparing the coordinates with element data in the BIM model, such as room numbers and floor information. A patient may be monitored to leave the ward and move to the emergency room. The system reflects this dynamic in real time on the BIM model and displays it to the medical team through a visual interface.

[0082] In some embodiments, the S4: based on the matching results, the digital twin model is updated in real time to reflect the real-time status of the building site, including: modifying the position and status of the elements corresponding to the people and objects in the digital twin model, which reflect the latest status of the people and objects identified in the video surveillance data. In some embodiments, the video data is analyzed to identify specific people and objects, such as customers, employees or security personnel, and important objects such as cleaning equipment or emergency medical kits. Through the point position matching algorithm, the system converts the two-dimensional coordinates in the video into three-dimensional space coordinates in the mall BIM model, and matches them to determine the specific location of each entity. Based on the matching results, the digital twin model of the mall is updated in real time to display the current location and status of each identified entity. For example, if a customer moves from the first floor to the second floor, the customer's position change will be reflected in real time on the digital twin model. At the same time, the model will also update the status of specific areas, such as identifying and marking the increase in traffic at the emergency exit in an emergency.

[0083] In some embodiments, the conversion of the identified coordinates or bounding box into the three-dimensional space coordinates of the BIM model includes converting the two-dimensional image coordinates into the three-dimensional space coordinates by a reverse projection method according to the calibration data of the camera, and the calibration data includes the position, orientation, focal length and lens distortion parameters of the camera. Converting the identified two-dimensional image coordinates into the three-dimensional space coordinates in the BIM model is a key technology in digital twin technology, ensuring the seamless integration of the video surveillance system with the physical building model. This process involves reverse projection using the calibration data of the camera to restore the three-dimensional world coordinates from the two-dimensional image plane coordinates. The reverse projection method is a calculation process based on the internal and external parameters of the camera, which can map the points on the image back to their positions in the real world. Camera calibration is the process of determining the internal and external parameters of the camera, which are necessary for accurate reverse projection. The internal parameters include focal length, optical center (usually the center point of the image), and lens distortion parameters (correcting the deformation effect caused by the lens). The external parameters describe the position and orientation of the camera relative to a known world coordinate system. The basic goal of reverse projection is to convert the two-dimensional image coordinates into three-dimensional space coordinates using the above parameters. Here are the specific steps: 2D image coordinate acquisition First identify and extract the coordinates or bounding box of the target (such as a person or object) from the video frame. Apply the camera intrinsic parameters to convert the 2D coordinates into normalized image plane coordinates using the focal length and optical center parameters. Apply the lens distortion parameters to correct the coordinates to correct any distortion caused by the optical properties of the lens. Use external parameters for spatial positioning: Use the position and orientation information of the camera to map the normalized image plane coordinates into a 3D world coordinate system. This usually involves projecting the 2D point onto a plane that passes through the optical center of the camera and at a certain angle to the camera, and then calculating the 3D coordinates based on the distance from the camera to the plane.

[0084] Embodiment 2:

[0085] This application also provides a digital twin positioning system based on BIM and video, such as Figure 2As shown, in some embodiments, the system hardware components include high-resolution cameras that can capture clear video streams. These cameras typically have night vision and dynamic range adjustment capabilities to adapt to various environmental conditions. They are installed in key locations of buildings, such as entrances, exits, main channels, etc., to fully monitor the activities of the venue. Servers and storage devices are used to process and store video data from cameras, while running data analysis software, such as deep learning models for image recognition and behavior analysis. High-performance servers are equipped with sufficient CPU, GPU and memory resources to support real-time video analysis and big data processing. Storage devices need to have sufficient capacity to store historical video data and its analysis results. Display devices include multiple display screens or interactive touch screens for displaying digital twin models and real-time videos, installed in control centers and key operating points, such as security monitoring rooms, management offices, etc. Network devices include routers, switches and wireless access points to ensure high-speed data transmission and network connections between cameras, servers and display devices. The camera is connected to a local server via high-speed Ethernet or Wi-Fi. These connections ensure that video data can be quickly and securely transmitted to the server for processing. The server is connected to each display device via a local area network (LAN). The server not only sends real-time video streams to these displays, but also sends visualization data of the digital twin model. The server is connected to a high-speed Internet connection for remote access and cloud data backup. This allows remote users to access system data and management functions, while also supporting remote backup and disaster recovery of data. Data Center or Cloud Platform For large-scale deployments, the server is connected to a larger data center or cloud platform to take advantage of additional computing and storage resources.

[0086] The acquisition module uses a camera to acquire a real-time first video stream of the construction site and pre-process the first video stream data;

[0087] The recognition module uses a recursive neural network (RNN) to process the collected first video stream to recognize people and objects in the video;

[0088] A serialization module performs frame serialization on the preprocessed real-time first video stream, decomposes the first video stream into a series of separate frames, and adjusts the frequency of extracting frame images according to a structural similarity index SSIM between frames;

[0089] The similarity index calculation module calculates the structural similarity index SSIM between consecutive frame pairs within the set time window and calculates the average value of the SSIM value within the set time window. avg ;

[0090] The extraction frequency determination module is based on the average value of the SSIM value in the time window. avg Adjust the video frame extraction frequency f of the current time window,

[0091] Among them, T max is the longest interval of video frame extraction; T min The shortest interval time for video frame extraction;

[0092] A second video stream forming module, splicing the extracted video frames to form a second video stream;

[0093] A second video stream feature extraction module uses a trained convolutional neural network CNN to perform feature extraction on the second video stream to extract features of the second video stream;

[0094] The frame sequence dynamic change feature extraction module inputs the extracted second video stream features into the recursive neural network RNN, and obtains the frame sequence dynamic change features through the recursive neural network RNN;

[0095] The recognition and classification module uses the softmax function to recognize and classify the dynamic change features of the frame sequence output by the recurrent neural network RNN ​​to obtain the people and objects in the video frame;

[0096] A position determination module matches the identified people and objects with element data in the BIM model, including using a point position matching algorithm to match the positions of people and objects identified in the video stream with rooms or floors in the BIM model;

[0097] The update module updates the digital twin model in real time based on the matching results to reflect the real-time status of the construction site;

[0098] The display module displays the updated digital twin model on the display device.

[0099] In some embodiments, the second video stream feature extraction module uses a trained convolutional neural network (CNN) to extract features from the second video stream to extract features of the second video stream; the second video stream features include edges and colors of people and objects; the trained convolutional neural network (CNN) uses an improved ModReLU activation function, and the improved ModReLU activation function is dynamically adjusted according to the video frame extraction frequency.

[0100] ModReLU(x,f)=max(0,x-β0*e -k*f );

[0101] Among them, β0 is the initial adjustment parameter, k is the exponential adjustment parameter, e is the base of the natural logarithm, f is the video frame extraction frequency, and x is the output of the convolution layer.

[0102] The present invention provides a digital twin positioning method and system based on BIM and video, which can achieve the following beneficial technical effects:

[0103] 1. The present invention first collects a real-time first video stream of a construction site with a camera and performs preprocessing, performs frame serialization on the preprocessed real-time first video stream, adjusts the frequency of extracting frame images according to the structural similarity index SSIM between frames to form a second video stream; uses a trained convolutional neural network CNN to extract features of the second video stream, extracts features of the second video stream, inputs the extracted features of the second video stream into a recursive neural network RNN, obtains dynamic change features of the frame sequence through the recursive neural network RNN, uses a softmax function to identify and classify the dynamic change features of the frame sequence output by the recursive neural network RNN, obtains people and objects in the video frame, matches the identified people and objects with the element data in the BIM model, and updates the digital twin model in real time according to the matching results to reflect the real-time status of the construction site. This application uses the average value of the SSIM value within the time window I avg The video frame extraction frequency f of the current time window is adjusted. During the feature extraction process, the improved ModReLU activation function is dynamically adjusted according to the video frame extraction frequency, which greatly improves the accuracy and efficiency of BIM model generation and greatly enhances the user experience.

[0104] 2. The present invention performs frame serialization on the preprocessed real-time first video stream, decomposes the first video stream into a series of separate frames, and adjusts the frequency of extracting frame images according to the structural similarity index SSIM between frames; calculates the structural similarity index SSIM between consecutive frame pairs within a set time window, and calculates the average value I of the SSIM value within the set time window. avg ; According to the average value of SSIM value in the time window I avg Adjust the video frame extraction frequency f of the current time window,

[0105] Among them, T max is the longest interval of video frame extraction; T min The shortest interval time of video frame extraction; This application calculates the structural similarity index SSIM between consecutive frame pairs, and calculates the average value of SSIM value in the set time window I avg Calculate the video frame extraction frequency. When the difference is large, it means that there are changes in people or objects. Increasing the extraction frequency greatly improves the accuracy of BIM model generation.

[0106] 3. The convolutional neural network CNN trained by the present invention adopts an improved ModReLU activation function, and the improved ModReLU activation function is dynamically adjusted according to the video frame extraction frequency.

[0107] ModReLU(x,f)=max(0,x-β0*e -k*f );

[0108] Among them, β0 is the initial adjustment parameter, k is the exponential adjustment parameter, e is the base of the natural logarithm, f is the video frame extraction frequency, and x is the output of the convolution layer; the present invention combines the video frame extraction frequency and adds it into the activation function calculation process. The improved activation function enables CNN to adjust its processing characteristics according to the dynamic changes of the video frame, enhances the flexibility and efficiency of processing video streams, and improves the adaptability and accuracy of the system to rapidly changing scenes. The video frame extraction frequency is integrated into the activation function of CNN as a dynamic parameter, so that the network can dynamically adjust its activation threshold according to the changes between frames when processing video frames, thereby more flexibly responding to changes in video content, greatly improving the accuracy and efficiency of BIM model position prediction.

[0109] The above is a detailed introduction to the digital twin positioning method and system based on BIM and video. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the core idea of ​​the present invention; at the same time, for those skilled in the art, according to the ideas and methods of the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A digital twin positioning method based on BIM and video, characterized in that: Includes steps: S1: The camera collects a real-time first video stream of the construction site and pre-processes the first video stream data; S2: using a recursive neural network (RNN) to process the collected first video stream to identify people and objects in the video; S21: performing frame serialization on the preprocessed real-time first video stream, decomposing the first video stream into a series of separate frames, and adjusting the frequency of extracting frame images according to a structural similarity index SSIM between frames; S211: Calculate the structural similarity index SSIM between consecutive frame pairs within the set time window, and find the average value I of the SSIM value within the set time window avg ; S212: According to the average value of SSIM value in the time window I avg Adjust the video frame extraction frequency f of the current time window, Among them, T max is the longest interval of video frame extraction; T min The shortest interval time for video frame extraction; S213: splicing the extracted video frames to form a second video stream; S22: using the trained convolutional neural network CNN to perform feature extraction on the second video stream, thereby extracting features of the second video stream; S23: inputting the extracted second video stream features into a recurrent neural network RNN, and obtaining dynamic change features of the frame sequence through the recurrent neural network RNN; S24: Using the softmax function to identify and classify the dynamic change features of the frame sequence output by the recurrent neural network RNN, and obtain the characters and objects in the video frame; S3: matching the identified people and objects with element data in the BIM model, including using a point position matching algorithm to match the positions of the people and objects identified in the video stream with the rooms or floors in the BIM model; based on the identified positions of the people and objects, that is, the coordinates or bounding boxes of the people and objects in the identified video frames; converting the identified coordinates or bounding boxes into three-dimensional space coordinates of the BIM model; and then determining the exact room or floor positions of the people and objects identified in the video in the BIM model; S4: Based on the matching results, the digital twin model is updated in real time to reflect the real-time status of the construction site; S5: Display the updated digital twin model on the display device.

2. The digital twin positioning method based on BIM and video according to claim 1, characterized in that: S1: The camera collects the real-time video stream of the construction site and pre-processes the video stream data, including using Gaussian filtering to denoise the video stream.

3. The digital twin positioning method based on BIM and video according to claim 1, characterized in that: S22: using the trained convolutional neural network CNN to extract features from the second video stream, extracting features of the second video stream; the features of the second video stream include edges and colors of people and objects; the trained convolutional neural network CNN uses an improved ModReLU activation function, and the improved ModReLU activation function is dynamically adjusted according to the video frame extraction frequency, ModReLU(x,f)=max(0,x-β0*e -k*f ); Among them, β0 is the initial adjustment parameter, k is the exponential adjustment parameter, e is the base of the natural logarithm, f is the video frame extraction frequency, and x is the output of the convolution layer.

4. The digital twin positioning method based on BIM and video according to claim 1, characterized in that: The S23: inputting the extracted second video stream features into the recursive neural network RNN, and obtaining the dynamic change features of the frame sequence through the recursive neural network RNN, including the recursive neural network calculating the current hidden state according to the second video stream features and the previous hidden state, thereby continuously updating the internal state to reflect the dynamic changes in the video frame sequence, and the recursive neural network extracting the dynamic change features of the frame sequence from the hidden state of each time step, the dynamic change features of the frame sequence including the motion trajectory, and the motion trajectory is marked with spatial coordinates.

5. The digital twin positioning method based on BIM and video according to claim 1, characterized in that: S24: using the softmax function to identify and classify the dynamic change features of the frame sequence output by the recursive neural network RNN, and obtaining the characters and objects in the video frame; applying the softmax function to process the dynamic change features of the frame sequence, and calculating the probability of obtaining the characters and objects, The one with the highest probability is the final classification result.

6. The digital twin positioning method based on BIM and video according to claim 1, characterized in that: S4: based on the matching results, updating the digital twin model in real time to reflect the real-time status of the construction site, including: modifying the position and status of elements corresponding to people and objects in the digital twin model, which reflect the latest status of people and objects identified in the video surveillance data.

7. The digital twin positioning method based on BIM and video according to claim 1, characterized in that: The converting of the identified coordinates or bounding box into three-dimensional space coordinates of the BIM model includes converting two-dimensional image coordinates into three-dimensional space coordinates by a back projection method according to calibration data of the camera, wherein the calibration data includes the position, orientation, focal length and lens distortion parameters of the camera.

8. The digital twin positioning system based on BIM and video is characterized by: include: The acquisition module uses a camera to acquire a real-time first video stream of the construction site and pre-process the first video stream data; The recognition module uses a recursive neural network (RNN) to process the collected first video stream to recognize people and objects in the video; A serialization module performs frame serialization on the preprocessed real-time first video stream, decomposes the first video stream into a series of separate frames, and adjusts the frequency of extracting frame images according to a structural similarity index SSIM between frames; The similarity index calculation module calculates the structural similarity index SSIM between consecutive frame pairs within a set time window and calculates the average value of the SSIM value within the set time window. avg ; The extraction frequency determination module is based on the average value of the SSIM value in the time window. avg Adjust the video frame extraction frequency f of the current time window, Among them, T max is the longest interval of video frame extraction; T min The shortest interval time for video frame extraction; A second video stream forming module, splicing the extracted video frames to form a second video stream; A second video stream feature extraction module uses a trained convolutional neural network CNN to perform feature extraction on the second video stream to extract features of the second video stream; The frame sequence dynamic change feature extraction module inputs the extracted second video stream features into the recursive neural network RNN, and obtains the frame sequence dynamic change features through the recursive neural network RNN; The recognition and classification module uses the softmax function to recognize and classify the dynamic change features of the frame sequence output by the recurrent neural network RNN ​​to obtain the people and objects in the video frame; The position determination module matches the identified people and objects with the element data in the BIM model, including using a point position matching algorithm to match the positions of people and objects identified in the video stream with the rooms or floors in the BIM model; based on the identified positions of people and objects, that is, the coordinates or bounding boxes of the people and objects in the identified video frames; converting the identified coordinates or bounding boxes into three-dimensional space coordinates of the BIM model; and then determining the exact room or floor positions of the people and objects identified in the video in the BIM model; The update module updates the digital twin model in real time based on the matching results to reflect the real-time status of the construction site; The display module displays the updated digital twin model on the display device.

9. The BIM and video-based digital twin positioning system according to claim 8, characterized in that: The second video stream feature extraction module uses a trained convolutional neural network (CNN) to perform feature extraction on the second video stream to extract features of the second video stream; the second video stream features include edges and colors of people and objects; the trained convolutional neural network (CNN) uses an improved ModReLU activation function, and the improved ModReLU activation function is dynamically adjusted according to the video frame extraction frequency. ModReLU(x,f)=max(0,x-β0*e -k*f ); Among them, β0 is the initial adjustment parameter, k is the exponential adjustment parameter, e is the base of the natural logarithm, f is the video frame extraction frequency, and x is the output of the convolution layer.

Images