Intelligent construction site safety management and control method and system based on multi-source data analysis

Abstract

The invention is suitable for the field of intelligent construction sites, and provides an intelligent construction site safety control method and system based on multi-source data analysis, and the method comprises the steps: employing a BERT model to carry out the feature extraction of a site information text and a weather information text, and obtaining text features; an improved SlowFast model is adopted to extract visual features of the image information; constructing a cross-modal relation matrix between the text and the visual features, carrying out weighted fusion on the text features and the visual features, and taking the fusion features as the input of an MLP network to obtain a preliminary behavior category of the worker in the image; comparing the preliminary behavior category result with a site feature and a weather feature, and adjusting the preliminary behavior category according to a predetermined correction rule to obtain a final behavior category; and based on the final behavior category, performing safety management and control on the behaviors of the construction site workers. According to the management and control method disclosed by the invention, comprehensive, accurate and real-time monitoring and management on the behaviors of the construction site workers can be realized through the acquisition and fusion of multi-source information and the combination of safety management and control measures.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent construction site management and control, and particularly relates to an intelligent construction site safety management and control method and system based on multi-source data analysis. Background Art

[0002] The internal environment of a construction site is complex and changeable. When construction workers operate inside the construction site, they not only need to operate in accordance with relevant regulations but also need to pay attention to the safety status of the area where they are located.

[0003] Currently, construction site safety management and control can reduce the probability of accidents to a certain extent. However, there are many deficiencies in traditional construction site safety monitoring means. Manual inspections are difficult to cover comprehensively and are easily affected by the quality and fatigue of personnel, while the warning effect of fixed safety notices is limited. This results in incomplete and discontinuous monitoring of construction workers, there are safety supervision loopholes, safety hazards cannot be discovered and warned in time, and the accuracy and reliability of construction site safety monitoring are reduced. Summary of the Invention

[0004] The purpose of the embodiments of the present invention is to provide an intelligent construction site safety management and control method and system based on multi-source data analysis, aiming to solve the technical problems proposed in the above background art.

[0005] To achieve the above purpose, the present invention provides the following technical solutions.

[0006] An embodiment of the present invention provides an intelligent construction site safety management and control method based on multi-source data analysis. The management and control method includes the following steps: S1. Obtain multi-source information of the construction site. The multi-source information includes site information, weather information, and image information; S2. Perform text preprocessing on the site information and weather information respectively, and use the BERT model to extract features from the preprocessed site information text and weather information text respectively to obtain site features and weather features. Concatenate the site features and weather features to obtain text features; S3. Use an improved SlowFast model to extract visual features of the image information; the improved SlowFast model includes a fast branch and a slow branch. A double-layer attention module is introduced into the fast branch, and a feature enhancement module is introduced into the slow branch. Adjust the weights of the fast branch and the slow branch based on a dynamic weight mechanism, and perform feature fusion on the outputs of the two branches according to the adjusted weights to obtain visual features; S4. Calculate the Pearson correlation coefficient to construct a cross-modal relationship matrix between the text and visual features, and weighted-fuse the text features and visual features according to the cross-modal relationship matrix to obtain the final fusion features; S5. Use the fused features as the input of the MLP network. Among them, introduce multi-stage hierarchical fusion residuals in the MLP network, extract features through residual MLP blocks in multiple stages, fuse shallow and deep features at each stage, and map the fused features to the corresponding behavior categories to obtain the preliminary behavior categories of the workers in the image. S6. Compare the preliminary behavior category results with the site features and weather features, and adjust the preliminary behavior categories according to the predetermined correction rules to obtain the final behavior categories. S7. Based on the final behavior categories, conduct safety control over the behaviors of the construction workers.

[0007] Further, in step S3, the steps of introducing a double-layer attention module in the fast branch include: S311. Perform a linear transformation on the input features to generate the query matrix, key matrix, and value matrix in the attention mechanism. S312. Construct a directed graph and establish the attention relationship between different regions, calculate the average values of the query matrix and key matrix in each region to generate the regional query matrix and regional key matrix, and generate the adjacency matrix by taking the dot product of the regional query matrix and regional key matrix. The adjacency matrix is used to measure the correlation between different regions. S313. Prune the adjacency matrix, including the adjacency matrix of the top k regions with high correlation, to obtain the routing index matrix. S314. Based on the attention mechanism concentrated on k routing regions, aggregate the key matrix and value matrix tensors of all routing regions to generate the aggregated key matrix and value matrix. S315. Perform an attention operation on the aggregated key matrix and value matrix, introduce the local context enhancement term LEC to derive the result tensor, and obtain the output features of the fast branch.

[0008] Further, in step S3, the feature enhancement module is a multi-branch structure, and the output feature maps of multiple branches are concatenated; the concatenated feature map and the input feature map are added through a residual connection and output through the ReLU activation function to obtain the output features of the slow branch.

[0009] Further, in the step of adjusting the weights of the fast branch and the slow branch based on the dynamic weight mechanism in step S3, the weight calculation formula is expressed as: ; Among them, represents the result of global average pooling of the feature sequence of the slow branch; represents the result of global average pooling of the feature sequence of the fast branch; conv1, conv2, and conv3 all represent convolution operations; σ represents the sigmoid activation function, which is used to limit the output result within the range of 0 to 1, and the output result is used to represent the size of the weight.

[0010] Further, in step S3, the step of performing feature fusion on the outputs of the two branches according to the adjusted weights to obtain visual features includes: Obtain the output features of the fast branch and the output features of the slow branch, denoted as and , where represents the output features of the slow branch, and represents the output features of the fast branch; Perform global average pooling on the feature sequences of the slow branch and the fast branch respectively to obtain the pooling results and ; Perform convolution operations on the two pooling results respectively and take the difference to calculate the motion feature difference F between the fast and slow branches, expressed as: ; where conv1 and conv2 both represent convolution operations, and σ represents the activation function; Perform a convolution operation on the motion feature difference F and use the sigmoid activation function to generate the feature weight , expressed as: ; where conv3 represents the convolution operation, and σ represents the sigmoid activation function; Perform a dot product operation on the feature weight and the features of the slow branch to generate an enhanced feature map , where ; Fuse the enhanced feature map with the features of the fast branch as the subsequent input of the slow branch to achieve the feature fusion of the fast and slow branches and obtain visual features.

[0011] Further, in step S4, the Pearson correlation coefficient is expressed as: ; In the formula, is the covariance of and ; is the standard deviation of ; is the standard deviation of ; is the mean of , is the mean of ; E represents the expected value.

[0012] Further, in step S4, the steps of constructing the cross-modal relationship matrix between the text and visual features include: Substitute the text features and visual features into the variables of the Pearson correlation coefficient respectively to construct a relationship fusion matrix, and obtain the relationship fusion matrix of text features and visual features , where The value of is the number of text features and visual features; the dimension of the matrix is ; the matrix reflects the mutual relationship between text features and visual features; Substitute the visual features and text features into the variables of the Pearson correlation coefficient to construct a relationship fusion matrix, and use the Pearson correlation coefficient to obtain the relationship matrix of visual features and text features .

[0013] Furthermore, in step S4, the step of weighted fusion of text features and visual features according to the cross-modal relationship matrix to obtain the final fusion features includes: After obtaining the relationship matrices and , weight the visual features and text features respectively; where: Weight the visual features through the text-to-visual weighted relationship matrix , expressed as: , represents the visual features generated after being weighted by the text-to-visual relationship matrix; Weight the text features through the visual-to-text weighted relationship matrix , expressed as: , represents the text features generated after being weighted by the visual-to-text weighted relationship matrix; and are the enhanced text features and visual features respectively; Finally, through the method of weighted average, fuse the enhanced text features and visual features together, expressed as: ; where is the final fusion feature vector, and are hyperparameters that control the contribution degrees of text features and visual features in the final fusion features.

[0014] Furthermore, in step S5, introduce multi-stage hierarchical fusion residuals in the MLP network, including: Divide the MLP network into multiple stages, each stage contains several residual MLP blocks, and the structure of each residual MLP block is expressed as: ; where represents the output of the layer, and MLP(·) represents a multi-layer perceptron; At the end of each stage, the output features of the current stage are fused with the output features of the previous stage, which is expressed as: ; where, represents the fused features of the s-th stage, and concat(·, ·) represents the feature concatenation operation; Skip connections are added between different stages of the network to fuse shallow features and deep features, which is expressed as: ; where, represents the output of the j-th skip connection, represents the transformation of the features; The fused features of the last stage are input into the classification layer, which is expressed as: ; where, O represents the final classification output, and W and b represent the weights and biases of the classification layer respectively.

[0015] Another embodiment of the present invention provides an intelligent construction site safety control system based on multi-source data analysis. The control system includes the following modules: A data acquisition module for acquiring multi-source information of the construction site. The multi-source information includes site information, weather information, and image information; A text feature extraction module for respectively performing text preprocessing on the site information and weather information, and using the BERT model to extract features from the preprocessed site information text and weather information text respectively to obtain site features and weather features, and performing concatenation processing on the site features and weather features to obtain text features; A visual feature extraction module for extracting visual features of the image information using an improved SlowFast model; the improved SlowFast model includes a fast branch and a slow branch. A double-layer attention module is introduced into the fast branch, and a feature enhancement module is introduced into the slow branch. The weights of the fast branch and the slow branch are adjusted based on a dynamic weight mechanism, and the outputs of the two branches are feature-fused according to the adjusted weights to obtain visual features; A feature fusion module for constructing a cross-modal relationship matrix between the text and visual features by calculating the Pearson correlation coefficient, and weighted-fusing the text features and visual features according to the cross-modal relationship matrix to obtain the final fused features; A behavior classification module for taking the fused features as the input of the MLP network. Among them, multi-stage hierarchical fusion residuals are introduced into the MLP network, features are extracted through residual MLP blocks in multiple stages, and shallow and deep features are fused in each stage, and the fused features are mapped to the corresponding behavior categories to obtain the preliminary behavior categories of the workers in the image; A behavior correction module, which is used to compare the preliminary behavior category results with the site characteristics and weather characteristics, and adjust the preliminary behavior category according to the predetermined correction rules to obtain the final behavior category; A behavior control module, which is used to perform safety control on the behaviors of construction workers based on the final behavior category.

[0016] Compared with the prior art, the beneficial effects of the intelligent construction site safety control method and system based on multi-source data analysis of the present invention are as follows: First, by integrating site information, weather information, and image information, the present invention can comprehensively understand the actual situation of the construction site. The BERT model is used to extract text features, enabling subsequent behavior recognition to consider the influence of the site and weather. By introducing a double attention module in the fast branch of the improved SlowFast model, it can better capture the motion information in the image sequence and improve the recognition accuracy of workers' behaviors. By introducing a feature enhancement module in the slow branch, it can enhance the ability to extract key features of workers using safety equipment. Based on the dynamic weight mechanism to adjust the weights of the fast branch and the slow branch, the model can adaptively balance the contributions of the two branches according to different inputs, thereby better integrating motion information and appearance information and improving the model's ability to capture different behavior features; Second, by calculating the Pearson correlation coefficient to construct a cross-modal relationship matrix, the present invention can quantify the correlation between text features and visual features, providing a basis for cross-modal feature fusion. By weighted fusing text features and visual features according to the relationship matrix, it can achieve deep fusion of multi-modal information, make full use of site and weather information to assist behavior recognition, and improve the accuracy and robustness of behavior recognition; Third, the present invention takes the fused features as the input of the MLP network. Through the improved MLP network, the multi-stage hierarchical fusion residual MLP enhances the adaptability of the network model to different feature patterns through feature fusion and residual connection, realizing the preliminary classification of workers' behaviors. Comparing the preliminary behavior category results with the site characteristics and weather characteristics and adjusting according to the predetermined correction rules can further improve the accuracy of behavior recognition. Site characteristics and weather characteristics provide the context information of the behavior occurrence, which helps to verify and correct the preliminary classification results and reduce misjudgments; In summary, the control method of the present invention can realize comprehensive, accurate, and real-time monitoring and management of the behaviors of construction workers through the acquisition and fusion of multi-source information and the combination of safety control measures. Brief Description of the Drawings

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention.

[0018] Figure 1 It is a flowchart for implementing the intelligent construction site safety control method based on multi-source data analysis of the present invention; Figure 2 It is a sub-flowchart of the intelligent construction site safety control method based on multi-source data analysis of the present invention; Figure 3 It is another sub-flowchart of the intelligent construction site safety control method based on multi-source data analysis of the present invention; Figure 4 It is yet another sub-flowchart of the intelligent construction site safety control method based on multi-source data analysis of the present invention; Figure 5 It is a structural block diagram of the intelligent construction site safety control system based on multi-source data analysis of the present invention; Figure 6 It is a structural block diagram of a computer device provided by the present invention. Detailed implementation manners

[0019] In order to make the objectives, technical solutions and advantages of the present invention more clear and understandable, the following further details the present invention in conjunction with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0020] The following details the specific implementation of the present invention in conjunction with specific embodiments.

[0021] Please refer to Figure 1 , in an embodiment of the present invention, an intelligent construction site safety control method based on multi-source data analysis is provided. The control method includes the following steps: S1. Obtain multi-source information of the construction site. The multi-source information includes site information, weather information, and image information; By integrating site information, weather information, and image information, the present invention can comprehensively understand the actual situation of the construction site. Among them, the site information mainly includes the layout of the construction site, including the specific locations and shapes of buildings, construction areas, material storage areas, and passageways. Through the site information, the functions of different areas can be clearly divided, such as high-altitude operation areas, foundation pit operation areas, welding operation areas, and general construction areas, etc., and at the same time, the safety levels and special requirements of each area are marked. The collection of weather information is achieved through two main channels: one is to call a high-precision weather forecast API to obtain weather forecast data for the next few hours to several days in the area where the construction site is located, including meteorological elements such as temperature, humidity, rainfall, wind force, wind direction, and air pressure; the other is to install reliable weather monitoring equipment, such as a weather station, at the construction site to monitor the actual weather conditions at the construction site in real time.

[0022] Furthermore, image information is a key data source for behavior recognition. High-definition and high-frame-rate cameras are installed in key areas and key parts of the construction site to ensure a reasonable layout of the cameras, which can comprehensively cover the main operation areas, passageways, and dangerous areas of the construction site, avoiding monitoring blind spots.

[0023] S2. Perform text preprocessing on the site information and weather information respectively, and use the BERT model to extract features from the preprocessed site information text and weather information text respectively to obtain site features and weather features, and perform splicing processing on the site features and weather features to obtain text features. The present invention uses the BERT model to extract text features, enabling subsequent behavior recognition to consider the influence of the site and weather. First, perform text preprocessing on the site information and weather information respectively to ensure the standardization and consistency of the data. The preprocessing process includes operations such as removing noise from the text, unifying the text format, and word segmentation. Then, use the BERT model to extract features from the preprocessed site information text and weather information text respectively. The BERT model captures rich semantic information and context relationships from the text. Specifically, the preprocessed site information text and weather information text are respectively input into the pre-trained BERT model. The BERT model will encode each word or sentence according to its internal multi-layer bidirectional Transformer structure to generate corresponding feature vectors, thereby obtaining site features and weather features. Finally, perform splicing processing on the site features and weather features to form a comprehensive text feature vector.

[0024] S3. Extract the visual features of the image information using the improved SlowFast model; the improved SlowFast model includes a fast branch and a slow branch. A double-layer attention module is introduced into the fast branch, and a feature enhancement module is introduced into the slow branch. The weights of the fast branch and the slow branch are adjusted based on the dynamic weight mechanism, and the outputs of the two branches are feature fused according to the adjusted weights to obtain the visual features; By introducing a double-layer attention module into the fast branch in combination with the improved SlowFast model, the present invention can better capture the motion information in the image sequence and improve the recognition accuracy of workers' behaviors; while introducing a feature enhancement module into the slow branch can enhance the ability to extract the key features of workers using safety equipment; based on the dynamic weight mechanism to adjust the weights of the fast branch and the slow branch, the present invention can enable the model to adaptively balance the contributions of the two branches according to different inputs, so as to better fuse the motion information and appearance information and improve the model's ability to capture different behavior features; S4. Calculate the Pearson correlation coefficient to construct a cross-modal relationship matrix between the text and the visual features, and fuse the text features and the visual features according to the cross-modal relationship matrix to obtain the final fused features; By calculating the Pearson correlation coefficient to construct a cross-modal relationship matrix, the present invention can quantify the correlation between the text features and the visual features, provide a basis for cross-modal feature fusion. Fusing the text features and the visual features according to the relationship matrix can achieve the deep fusion of multi-modal information, make full use of the site and weather information to assist in behavior recognition, and improve the accuracy and robustness of behavior recognition; S5. Use the fused features as the input of the MLP network. Among them, multi-stage hierarchical fusion residuals are introduced into the MLP network, features are extracted through multiple stages of residual MLP blocks, and shallow and deep features are fused at each stage, and the fused features are mapped to the corresponding behavior categories to obtain the preliminary behavior categories of the workers in the image; S6. Compare the preliminary behavior category results with the site features and weather features, and adjust the preliminary behavior categories according to the predetermined correction rules to obtain the final behavior categories; S7. Based on the final behavior categories, conduct safety control over the behaviors of the construction site workers; By using the fused features as the input of the MLP network, the present invention, through the improved MLP network, uses multi-stage hierarchical fusion residuals MLP to enhance the adaptability of the network model to different feature patterns through feature fusion and residual connection, and realizes the preliminary classification of workers' behaviors; Furthermore, comparing the preliminary behavior category results with the site features and weather features and adjusting according to the predetermined correction rules can further improve the accuracy of behavior recognition; The site characteristics and weather characteristics provide the context information for the occurrence of behaviors, which helps to verify and correct the preliminary classification results and reduce misjudgments.

[0025] In the embodiments of the present invention, by combining the position of the worker with the site information, when controlling the behavior of the worker, the specific area where the worker is currently located can be determined. For example, whether the worker is in the high-altitude operation area, near the foundation pit, in the tower crane operation area or the ordinary construction passage, etc. Different areas correspond to different types of possible behaviors and different safety requirements, which provides key context information for spatial perception and behavior positioning; in addition, according to the functional division of the site, the behavior of the worker in certain areas can be reasonably expected. For example, in the material stacking area, it is normal for the worker to have the behavior of handling materials, but if the worker has behaviors related to material processing such as welding or cutting in this area, there may be certain safety hazards.

[0026] Therefore, in the embodiments of the present invention, after the site information is fused with multi-source information such as weather information and image information, more comprehensive context can be provided. For example, in a high-altitude operation area with strong wind weather, if the worker shows behaviors such as body tilting, combining the site information (high-altitude operation area) and weather information (strong wind), it can be more accurately judged that this is a normal reaction of the worker to maintain balance in the strong wind environment, rather than a dangerous and illegal behavior. This kind of multi-source information fusion enables behavior recognition to fully consider site factors, thereby improving the accuracy of recognition.

[0027] In step S6 of the present invention, by comparing the preliminary behavior category results with the site characteristics and weather characteristics, based on the comparison results, the behavior category results are revised; here it is considered that the construction site is a complex environment with various uncertainties; the weather may also change suddenly, and the behavior patterns of workers will change greatly, such as the walking speed slows down, and the operation actions may be more cautious, etc.; therefore, the subsequent behavior correction steps of the present invention can further correct the behavior recognition results by considering more factors such as the trend and change speed of weather changes.

[0028] Please refer to Figure 2 , in step S3 of the embodiments of the present disclosure, the step of introducing a double attention module into the fast branch includes: S311. Linearly transform the input features to generate the query matrix, key matrix and value matrix in the attention mechanism; S312. Construct a directed graph and establish the attention relationship between different regions, calculate the average values of the query matrix and key matrix in each region to generate the region query matrix and region key matrix, and generate the adjacency matrix by taking the dot product of the region query matrix and region key matrix. The adjacency matrix is used to measure the correlation between different regions; S313. Prune the adjacency matrix, including the adjacency matrices of the top k highly correlated regions, to obtain a routing index matrix; S314. Based on the attention mechanism concentrated on k routing regions, aggregate the key matrix and value matrix tensors of all routing regions to generate an aggregated key matrix and value matrix; S315. Perform an attention operation on the aggregated key matrix and value matrix, introduce a local context enhancement term LEC to derive a result tensor, and obtain the output features of the fast branch.

[0029] Further, in step S3, the feature enhancement module is a multi-branch structure, and the output feature maps of multiple branches are concatenated; the concatenated feature map and the input feature map are added through a residual connection and output through a ReLU activation function to obtain the output features of the slow branch; in one implementation of the multi-branch structure, a three-branch structure is adopted, specifically including: The first branch with two convolutional layers. One convolutional layer uses a 1×1 convolutional kernel with a stride of stride to adjust the number of channels of the input feature map to 2×inter_planes of the original; the other convolutional layer uses a 3×3 convolutional kernel with a stride of 1 and a padding of 1 to extract local features. The first branch can increase the number of channels of the features and extract richer local features without changing the size of the feature map. The second branch with four convolutional layers. One convolutional layer uses a 1×1 convolutional kernel with a stride of 1 to adjust the number of channels of the input feature map to inter_planes; another convolutional layer uses a 1×3 convolutional kernel with a stride of stride and a padding of (0,1) to expand the receptive field in the height direction; the last convolutional layer uses a 3×1 convolutional kernel with a stride of stride and a padding of (1,0) to expand the receptive field in the width direction; the fourth convolutional layer uses a 3×3 convolutional kernel with a stride of 1, a padding of 5, and a dilation rate of 5 to further extract dilated convolutional features, expand the receptive field, and capture more context information. The third branch with three convolutional layers. One convolutional layer uses a 1×1 convolutional kernel with a stride of stride to adjust the number of channels of the input feature map to 2×inter_planes of the original; another convolutional layer uses a 3×1 convolutional kernel with a stride of stride and a padding of (1,0), and the last convolutional layer uses a 1×3 convolutional kernel with a stride of stride and a padding of (0,1) to extract features from different convolutional kernel dimension orders and enrich the diversity of features. Further, concatenate the output feature maps of the three branches; adjust the number of channels to out_planes through a convolutional layer, and finally add it to the input feature map through a residual connection and output through a ReLU activation function to obtain the output features of the slow branch.

[0030] In the embodiment of the present invention, in the step of adjusting the weights of the fast branch and the slow branch based on the dynamic weight mechanism provided in step S3, the weight calculation formula is expressed as: ; where represents the result of global average pooling of the feature sequence of the slow branch; represents the result of global average pooling of the feature sequence of the fast branch; conv1, conv2, and conv3 all represent convolution operations; σ represents the sigmoid activation function, which is used to limit the output result within the range of 0 to 1, and the output result is used to represent the size of the weight.

[0031] As Figure 3 shown, in step S3, the step of performing feature fusion on the outputs of the two branches according to the adjusted weights to obtain visual features includes: S321. Obtain the output features of the fast branch and the slow branch, denoted as and , respectively, where represents the output feature of the slow branch, represents the output feature of the fast branch; S323. Perform global average pooling on the feature sequences of the slow branch and the fast branch respectively to obtain the pooling results and ; S323. Perform convolution operations on the two pooling results and take the difference to calculate the motion feature difference F between the fast and slow branches, which is expressed as: ; where conv1 and conv2 both represent convolution operations, and σ represents the activation function; S324. Perform a convolution operation on the motion feature difference F and use the sigmoid activation function to generate the feature weight , which is expressed as: ; where conv3 represents the convolution operation, and σ represents the sigmoid activation function; S325. Perform a dot product operation on the feature weight and the feature of the slow branch to generate the enhanced feature map , where ; S326. Fuse the enhanced feature map with the feature of the fast branch as the subsequent input of the slow branch to achieve feature fusion between the fast and slow branches and obtain visual features.

[0032] Furthermore, in step S4, the Pearson correlation coefficient is expressed as: ; In the formula, is the covariance of ; is the standard deviation of is the standard deviation of is the mean of is the mean of; E represents the expected value.

[0033] Furthermore, in step S4, the step of constructing the cross-modal relationship matrix between text and visual features includes: substituting text features and visual features into the variables of the Pearson correlation coefficient respectively to construct a relationship fusion matrix, and obtaining the relationship fusion matrix of text features and visual features , where the value of is the number of text features and visual features; the dimension of the matrix is ; the matrix reflects the mutual relationship between text features and visual features; substituting visual features and text features into the variables of the Pearson correlation coefficient to construct a relationship fusion matrix, and using the Pearson correlation coefficient, the relationship matrix of visual features and text features can be obtained .

[0034] In step S4 of the present invention, the step of weighted fusing text features and visual features according to the cross-modal relationship matrix to obtain the final fused features includes: after obtaining the relationship matrix and the relationship matrix , respectively weighting visual features and text features; where: weighting visual features through the text-to-visual weighted relationship matrix , expressed as: , represents the visual features generated after weighting by the text-to-visual relationship matrix; weighting text features through the visual-to-text weighted relationship matrix , expressed as: , represents the text features generated after weighting by the visual-to-text weighted relationship matrix; where and are the enhanced text features and visual features respectively; finally, by the method of weighted average, fusing the enhanced text features and visual features together, expressed as: ; where is the final fused feature vector, and are hyperparameters that control the contribution degrees of text features and visual features in the final fused features.

[0035] Furthermore, please refer toFigure 4 , in step S5, a multi-stage hierarchical fusion residual is introduced into the MLP network, including: S411. Divide the MLP network into multiple stages, each stage contains several residual MLP blocks, and the structure of each residual MLP block is expressed as: ; where represents the output of the -th layer, and MLP(·) represents a multi-layer perceptron; S412. At the end of each stage, fuse the output features of the current stage with the output features of the previous stage, which is expressed as: ; where represents the fused feature of the s-th stage, and concat(·, ·) represents a feature concatenation operation; S413. Add skip connections between different stages of the network to fuse shallow features and deep features, which is expressed as: ; where represents the output of the j-th skip connection, represents a transformation of the features; S414. Input the fused feature of the last stage into the classification layer, which is expressed as: ; where O represents the final classification output, and W and b represent the weights and biases of the classification layer respectively.

[0036] In summary, the control method of the present invention can realize comprehensive, accurate and real-time monitoring and management of the behaviors of construction site workers through the acquisition and fusion of multi-source information and the combination of safety control measures.

[0037] Please refer to Figure 5 , in another embodiment of the present invention, a smart construction site safety control system based on multi-source data analysis is provided. The control system includes the following modules: A data acquisition module 81, which is used to acquire multi-source information of the construction site. The multi-source information includes site information, weather information and image information; A text feature extraction module 82, which is used to perform text preprocessing on the site information and weather information respectively, and use the BERT model to extract features from the preprocessed site information text and weather information text respectively to obtain site features and weather features, and perform concatenation processing on the site features and weather features to obtain text features; A visual feature extraction module 83, which is used to extract visual features of the image information by using an improved SlowFast model; the improved SlowFast model includes a fast branch and a slow branch. A double-layer attention module is introduced into the fast branch, and a feature enhancement module is introduced into the slow branch. The weights of the fast branch and the slow branch are adjusted based on a dynamic weight mechanism, and the outputs of the two branches are feature-fused according to the adjusted weights to obtain visual features; The feature fusion module 84 is used to construct a cross-modal relationship matrix between text and visual features by calculating the Pearson correlation coefficient, and weighted-fuse the text features and visual features according to the cross-modal relationship matrix to obtain the final fused features; The behavior classification module 85 is used to take the fused features as the input of the MLP network. Among them, multi-stage hierarchical fusion residuals are introduced into the MLP network, features are extracted through residual MLP blocks in multiple stages, and shallow and deep features are fused at each stage, and the fused features are mapped to the corresponding behavior categories to obtain the preliminary behavior categories of the workers in the image; The behavior correction module 86 is used to compare the preliminary behavior category results with the site features and weather features, and adjust the preliminary behavior categories according to the predetermined correction rules to obtain the final behavior categories; The behavior control module 87 is used to perform safety control on the behaviors of the construction site workers based on the final behavior categories.

[0038] As Figure 6 shown, the computer device includes a processor, a memory, a network interface, an input device, and a display screen connected through a system bus. Among them, the memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the computer device stores an operating system and can also store a computer program. When the computer program is executed by the processor, the processor can implement the intelligent construction site safety control method based on multi-source data analysis.

[0039] The internal memory can also store a computer program. When the computer program is executed by the processor, the processor can execute the intelligent construction site safety control method based on multi-source data analysis.

[0040] Those skilled in the art can understand that Figure 6 the structure shown in

[0041] is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0042] It should be understood that although the steps in the flowcharts of the embodiments of the present invention are shown in sequence according to the indications of the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear indication in this article, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of the steps in each embodiment may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or sub-steps or stages of other steps.

[0043] Those of ordinary skill in the art can understand that all or part of the processes of implementing the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memories. Non-volatile memories can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memories can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0044] The technical features of the above-described embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this specification.

[0045] The above-described embodiments merely represent several implementation manners of the present invention. The description thereof is relatively specific and detailed, but it should not be construed as a limitation to the scope of the patent for the present invention. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several variations and improvements can still be made, and these all fall within the protection scope of the present invention. Therefore, the protection scope of the patent for the present invention shall be subject to the appended claims.

[0046] The foregoing is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent replacements, and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A smart construction site safety management and control method based on multi-source data analysis, characterized in that: The control method includes the following steps: S1. Acquire multi-source information of the construction site, including site information, weather information and image information; S2. Preprocess the venue information and weather information respectively, and use the BERT model to extract features from the preprocessed venue information text and weather information text respectively to obtain venue features and weather features, and concatenate the venue features and weather features to obtain text features; S3, using an improved SlowFast model to extract visual features of image information; the improved SlowFast model includes a fast branch and a slow branch, a double-layer attention module is introduced in the fast branch, a feature enhancement module is introduced in the slow branch, the weights of the fast branch and the slow branch are adjusted based on a dynamic weight mechanism, and the outputs of the two branches are feature fused according to the adjusted weights to obtain visual features; S4. By calculating the Pearson correlation coefficient, a cross-modal relationship matrix between text and visual features is constructed, and the text features and visual features are weightedly fused according to the cross-modal relationship matrix to obtain the final fusion features; S5. The fused features are used as the input of the MLP network, wherein a multi-stage hierarchical fusion residual is introduced into the MLP network, features are extracted through residual MLP blocks of multiple stages, shallow and deep features are fused at each stage, and the fused features are mapped to the corresponding behavior categories to obtain the preliminary behavior categories of the workers in the image; S6, comparing the preliminary behavior category results with the site characteristics and weather characteristics, and adjusting the preliminary behavior category according to a predetermined correction rule to obtain a final behavior category; S7. Based on the final behavior category, safety management and control of the behavior of construction site workers is carried out.

2. The smart construction site safety management and control method based on multi-source data analysis according to claim 1 is characterized in that: In step S3, the step of introducing a double-layer attention module into the fast branch includes: S311, linearly transform the input features to generate a query matrix, a key matrix, and a value matrix in the attention mechanism; S312, construct a directed graph and establish attention relationships between different regions, calculate the average values ​​of the query matrix and the key matrix in each region, generate a region query matrix and a region key matrix, and generate an adjacency matrix by performing a dot product of the region query matrix and the region key matrix. The adjacency matrix is ​​used to measure the correlation between different regions; S313, pruning the adjacency matrix, including the first k high-correlation region adjacency matrices, to obtain a routing index matrix; S314, based on the attention mechanism focused on the k routing areas, aggregating the key matrix and value matrix tensors of all routing areas to generate an aggregated key matrix and value matrix; S315. Perform an attention operation on the aggregated key matrix and value matrix, introduce a local context enhancement term LEC to derive the result tensor, and obtain the output features of the fast branch.

3. The smart construction site safety management and control method based on multi-source data analysis according to claim 2 is characterized in that: In step S3, the feature enhancement module has a multi-branch structure, and the output feature maps of multiple branches are spliced; the spliced ​​feature map is added to the input feature map through a residual connection, and the output feature of the slow branch is obtained through the ReLU activation function output.

4. The smart construction site safety management and control method based on multi-source data analysis according to claim 3 is characterized in that: In step S3, in the step of adjusting the weights of the fast branch and the slow branch based on the dynamic weight mechanism, the weight calculation formula is expressed as: ; in, Indicates the result of global average pooling of the feature sequence of the slow branch; It represents the result of global average pooling of the feature sequence of the fast branch; conv1, conv2, and conv3 all represent convolution operations; σ represents the sigmoid activation function, which is used to limit the output result to the range of 0 to 1. The output result is used to represent the size of the weight.

5. The smart construction site safety management and control method based on multi-source data analysis according to claim 4 is characterized in that: In step S3, the outputs of the two branches are subjected to feature fusion according to the adjusted weights to obtain visual features, including: Get the output characteristics of the fast branch and the output characteristics of the slow branch, which are recorded as and ,in, represents the output characteristics of the slow branch, Indicates the output characteristics of the fast branch; Perform global average pooling on the feature sequences of the slow branch and the fast branch respectively to obtain the pooling results and ; The two pooling results are convolved and subtracted to calculate the motion feature difference F between the fast and slow branches, which is expressed as: ; Where conv1 and conv2 both represent convolution operations, and σ represents the activation function; Perform convolution operation on the motion feature difference F and use sigmoid activation function to generate feature weights , expressed as: ; Where conv3 represents the convolution operation and σ represents the sigmoid activation function; The feature weights are combined with the features of the slow branch Perform point multiplication to generate enhanced feature maps ,in, ; The enhanced feature map The features of the fast branch are fused together as the subsequent input of the slow branch to achieve feature fusion of the fast and slow branches and obtain visual features.

6. The smart construction site safety management and control method based on multi-source data analysis according to claim 5 is characterized in that: In step S4, the Pearson correlation coefficient It is expressed as: ； In the formula, yes and The covariance of yes The standard deviation of yes The standard deviation of yes The mean of yes E represents the expected value.

7. The smart construction site safety management and control method based on multi-source data analysis according to claim 6 is characterized in that: In step S4, the step of constructing a cross-modal relationship matrix between text and visual features includes: Substitute the text features and visual features into the variables of the Pearson correlation coefficient respectively, construct the relationship fusion matrix, and obtain the relationship fusion matrix between text features and visual features ,in, The value of is the number of text features and visual features; the dimension of the matrix is ;matrix It reflects the relationship between text features and visual features; Substitute the visual features and text features into the variables of the Pearson correlation coefficient to construct a relationship fusion matrix. Using the Pearson correlation coefficient, the relationship matrix between visual features and text features can be obtained. 。 8. The smart construction site safety management and control method based on multi-source data analysis according to claim 7 is characterized in that: In step S4, the step of weighting and fusing text features and visual features according to the cross-modal relationship matrix to obtain the final fusion features includes: In the relationship matrix and the relationship matrix After that, the visual features and text features are weighted respectively; where: Weighted Relationship Matrix via Text-to-Vision To weight the visual features, it is expressed as: , Represents the visual features generated by weighting the text-to-visual relationship matrix; Vision-to-text weighted relationship matrix To weight the text features, it is expressed as: , Represents the text features generated by weighting the visual-to-text weighted relationship matrix; and They are the enhanced text features and visual features respectively; Finally, the enhanced text features and visual features are fused together through the weighted average method, expressed as: ;in, is the final fused feature vector, and It is a hyperparameter that controls the contribution of text features and visual features in the final fusion features.

9. The smart construction site safety management and control method based on multi-source data analysis according to claim 8 is characterized in that: In step S5, a multi-stage hierarchical fusion residual is introduced into the MLP network, including: The MLP network is divided into multiple stages, each stage contains several residual MLP blocks, and the structure of each residual MLP block is expressed as: ； in, Indicates The output of the layer, MLP (·) represents a multi-layer perceptron; At the end of each stage, the output features of the current stage are fused with the output features of the previous stage, expressed as: ;in, represents the fusion feature of the sth stage, and concat(·,·) represents the feature concatenation operation; Adding skip connections between different stages of the network to fuse shallow features and deep features is expressed as: ;in, represents the output of the j-th skip connection, Indicates the transformation of features; The fused features of the last stage are input into the classification layer, expressed as: ; Where O represents the final classification output, W and b represent the weight and bias of the classification layer respectively.

10. A control system for implementing the smart construction site safety control method based on multi-source data analysis as claimed in any one of claims 1 to 9, characterized in that: The management and control system includes the following modules: A data acquisition module is used to acquire multi-source information of the construction site, including site information, weather information and image information; A text feature extraction module is used to perform text preprocessing on the venue information and weather information respectively, and use the BERT model to extract features from the preprocessed venue information text and weather information text respectively to obtain venue features and weather features, and then perform splicing processing on the venue features and weather features to obtain text features; A visual feature extraction module is used to extract visual features of image information using an improved SlowFast model; The improved SlowFast model includes a fast branch and a slow branch. A double-layer attention module is introduced in the fast branch, and a feature enhancement module is introduced in the slow branch. The weights of the fast branch and the slow branch are adjusted based on a dynamic weight mechanism. The outputs of the two branches are feature fused according to the adjusted weights to obtain visual features. The feature fusion module is used to construct a cross-modal relationship matrix between text and visual features by calculating the Pearson correlation coefficient, and to weight the text features and visual features according to the cross-modal relationship matrix to obtain the final fusion features; The behavior classification module is used to use the fused features as the input of the MLP network. In the MLP network, multi-stage hierarchical fusion residuals are introduced to extract features through residual MLP blocks in multiple stages, and shallow and deep features are fused at each stage. The fused features are mapped to the corresponding behavior categories to obtain the preliminary behavior categories of the workers in the image; A behavior correction module is used to compare the preliminary behavior category results with the site characteristics and weather characteristics, and adjust the preliminary behavior category according to a predetermined correction rule to obtain a final behavior category; The behavior control module is used to conduct safety control on the behavior of construction site workers based on the final behavior category.

Images