Graph-text multi-mode intelligent identification method, system and equipment for potential safety hazards in hydropower engineering construction

Abstract

The invention belongs to the technical field of hydropower engineering construction safety intelligent monitoring, and particularly provides a hydropower engineering construction potential safety hazard image-text multi-mode intelligent identification method, system and equipment, and the method comprises the steps: collecting hydropower engineering construction potential safety hazard text and image data, and building a corresponding relation between a potential hazard text and a potential hazard image; processing the hidden danger text by using a BERT model, extracting text features, processing hidden danger image data by using a Vision Transform model of a multi-head self-attention mechanism, and extracting image features; fusing the extracted text features and image features through Gate Net to obtain fused feature expression; a feature with a higher level is extracted from the fused features through a Transform model; and classifying the processed features by using a multi-layer sensor, and outputting results of various hidden dangers. According to the method, association between a construction potential safety hazard text and an image is fully considered, an image-text multi-modal joint characterization principle based on gating circulation fusion is provided, and a construction potential safety hazard multi-modal classification identification method is provided.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of intelligent monitoring of water and electricity engineering construction safety, and in particular relates to a water and electricity engineering construction safety hidden danger image-text multi-modal intelligent identification method, system and equipment. BACKGROUND

[0002] The water and electricity engineering construction environment is complex and changeable, involving high slope excavation, large-scale mechanical operation, high-altitude operation, and temporary power supply and other high-risk scenes, and its safety hidden danger has the characteristics of diversity, concealment and dynamics. Compared with general building engineering, water and electricity engineering is often located in remote mountainous areas or valley areas, with complex geological conditions, long construction period, and significant influence of natural factors such as weather and hydrology, which makes safety management face greater challenges. The complex construction environment may inevitably bring noise to image acquisition, such as light changes and complex backgrounds, and it is difficult to obtain satisfactory recognition results only relying on low-level visual features in images. Once a safety accident occurs, it is easy to cause significant casualties and property losses, and even trigger secondary disasters. The traditional safety hidden danger identification mainly relies on expert experience, and is judged by manual inspection or image data analysis. This method has obvious defects such as strong subjectivity and low efficiency. The identification result is affected by the personal experience and cognitive level of the expert, and it is difficult to standardize. Manual inspection is difficult to cover large-scale construction sites, and cannot monitor dynamic risks in real time. At the same time, relying on professional safety personnel leads to large human input and is difficult to sustain for a long time.

[0003] According to the standards of GB6441-86 and SL721-2015, the hidden dangers in the construction site of the water and electricity engineering mainly include 12 typical hidden danger categories such as falling from a height, object impact, mechanical injury, electric shock accident, traffic accident, explosion accident, lifting injury, fire accident, collapse accident, civilized construction, equipment damage and other accidents. These hidden dangers have the dual characteristics that standardized safety management can significantly reduce the accident rate, but once out of control, it will trigger equipment damage, personnel casualties and even secondary disasters such as collapse. The traditional intelligent identification method has obvious limitations. Although single image recognition technology can detect visually dominant hidden dangers, it is difficult to analyze the implicit semantic information in text inspection. Although pure text mining algorithms can extract hidden danger keywords from structured reports, they cannot fully understand the visual association of hidden dangers in the construction site. This mode of analysis, combined with subjective dependence on expert experience, leads to a high risk of misjudgment in hidden danger identification. It is urgent to develop a water and electricity engineering construction safety hidden danger intelligent identification method based on image-text multi-modal fusion to fully utilize the complementary information of images and texts and improve the accuracy and robustness of hidden danger identification. SUMMARY

[0004] The technical problem to be solved by the present application is to provide a hydropower engineering construction safety hazard graphic-text multi-modal intelligent identification method, system and equipment, which fully considers the correlation between construction safety hazard text and image, and proposes a graphic-text multi-modal intelligent identification method based on a gate fusion network, aiming at the problems of incomplete single-modal feature expression and low graphic-text feature fusion efficiency in hydropower engineering construction safety hazard identification.

[0005] To solve the above technical problems, the technical solution adopted by the present application is: a hydropower engineering construction safety hazard graphic-text multi-modal intelligent identification method, comprising the following steps: Step one, data input: collect hydropower engineering construction safety hazard text and image data, and establish a corresponding relationship between hazard text and hazard image; Step two, feature extraction: process the hazard text using a BERT model to extract text features, and process the hazard image data using a multi-head self-attention mechanism Vision Transformer model to extract image features; Step three, feature fusion: fuse the extracted text features and image features through a gate recurrent network Gate Net to obtain a fused feature expression; Step four, feature processing: further process the fused features through a Transformer model to extract higher-level features; Step five, hazard classification: classify the processed features using a multilayer perceptron to output the results of each type of hazard.

[0006] In the preferred scheme, after collecting the hydropower engineering construction safety hazard text and image data in step one, the text multi-labels and invalid images are removed, the data set is preprocessed, including setting labels, dividing, and establishing the corresponding relationship between the hazard text and the hazard image.

[0007] In the preferred scheme, in step two, the processed hazard text is input into the BERT model to obtain a 768-dimensional vector representation, and the implementation formula is: (1); (2); Wherein, w n represents the words in the hazard text description statement; t i represents the initial sequence of the hazard text; represents the hyperparameters of the BERT model; represents the vector space; is the vector representation output after the hazard text is encoded by the BERT model; is the encoding operation process of the BERT model, which is a function of mapping text sequences to high-dimensional vectors.

[0008] In a preferred scheme, in step two, the hidden danger image data is processed using the multi-head self-attention mechanism Vision Transformer model to extract image features. In the multi-head self-attention mechanism Vision Transformer model, patch is a small size image block obtained by uniformly cutting the original hidden danger image, token is a vector representation of the image patch or a special mark, and is the basic unit of Transformer processing. The specific operation of step two is as follows: The image is cut into small blocks patch as the input of the model. Before the input is delivered to the encoder layer, a token for classification is added, and a token vector for identifying classification is inserted at the beginning of each sequence. For each input token, three vectors , and are generated after input cross feature fusion. (3) (4) wherein the function of soft max(·) is normalization, denotes scaling of the feature; denotes the transpose of the vector ; denotes the patch of the hidden danger image converted into a token sequence to generate a self-attention calculation function; denotes the multi-head self-attention encoding process; is the vector representation of the hidden danger image output after encoding by the Vision Transformer model; denotes the connection of multiple vectors in the feature dimension; denotes an independent self-attention unit.

[0009] In a preferred scheme, in step three, the extracted text features and image features are fused by the gate recurrent network Gate Net for feature expression, and the operation is as follows: The modal representation based on the text hidden danger is f text and the modal representation based on the image hidden danger is f img as input, and the output gates of the text and image are obtained by inputting into the gate recurrent network Gate Net, and the calculation formula is as follows: (5); (6); wherein, is the output gate of text; is the output gate of image.

[0010] In the preferred scheme, in the step five, the hidden danger classification specifically comprises the following steps: S501, utilize the Gate Net to solve the interaction and screening of the image-text multi-modal information, so that the information between different modalities is fully interacted, and the redundant and irrelevant information is eliminated through the gating mechanism, and then the fused features are input into the multi-layer perceptron for nonlinear mapping and discrimination, and finally the category probability distribution is output: (7); (8); wherein, x denotes the multi-modal feature vector to be processed in the Gate Net; denotes an activation function, which maps the input to the interval [0, 1], and the output is used as a gating weight; denotes a multi-layer perceptron, which performs nonlinear feature transformation on the input x ; S502, adopt a cross-entropy loss function to supervise the learning of the category probability prediction model, and optimize the objective function J denotes: (9); wherein, k is the total number of categories, y t is the true label of the first t sample, if the sample belongs to the category t , t , then y t =1, otherwise 0; P t is the corresponding predicted probability value.

[0011] In the preferred scheme, in the step five, the results of each type of hidden danger include 12 types of hidden danger types, including explosion accident, vehicle injury, electric shock accident, falling from height, fire accident, mechanical injury, object strike, hoisting injury, collapse accident, equipment damage, civilization accident and other accidents.

[0012] The application also provides a water and electricity engineering construction safety hidden danger image-text multi-modal intelligent identification system for the above-mentioned identification method, comprising: The data input module is used for collecting hydropower engineering construction safety hidden danger text and image data, and establishing a corresponding relationship between the hidden danger text and the hidden danger image. The feature extraction module processes the hidden danger text by using a BERT model, extracts text features, processes the hidden danger image data by using a multi-head self-attention mechanism Vision Transformer model, and extracts image features. The feature fusion module fuses the extracted text features and image features through a gate recurrent network Gate Net to obtain a fused feature expression. The feature processing module further processes the fused features through a Transformer model to extract higher-level features. The hidden danger classification module classifies the processed features by using a multilayer perceptron and outputs the results of various hidden dangers.

[0013] The application also provides a computer readable storage medium, which stores computer instructions, and the computer instructions are used to make the computer execute the above-mentioned hydropower engineering construction safety hidden danger image-text multi-modal intelligent identification method.

[0014] The application also provides an electronic device, which comprises a memory and a processor, the memory and the processor are communicatively connected, the memory stores computer instructions, and the processor executes the computer instructions to execute the above-mentioned hydropower engineering construction safety hidden danger image-text multi-modal intelligent identification method.

[0015] The application provides a hydropower engineering construction safety hidden danger image-text multi-modal intelligent identification method, system and device, which has the following beneficial effects: 1. The method establishes exclusive classification principles for 12 kinds of construction safety hidden danger features according to the characteristics of hydropower engineering construction, accurately matches the hidden danger features of high-risk scenes of hydropower engineering, proposes a construction safety hidden danger image-text multi-modal joint representation method, realizes cross-modal accurate alignment of hidden danger features, and solves the industry pain points of "easy to miss judgment of explicit features and difficult to capture implicit risks" in traditional identification.

[0016] 2. The method invents a quick and easy-to-operate construction safety hidden danger image-text multi-modal intelligent identification method, innovatively uses a gate recurrent network to realize dynamic screening and interactive fusion of text and image features, adaptively adjusts the information transmission ratio of images and texts, solves the problems of incomplete feature expression and poor robustness in traditional single-modal identification. The implementation case data shows that, compared with a pure image model, the recall rate of the gate fusion model is improved by 23.26%, the F1 score is improved by 12.84%, and the gate fusion model is significantly better than existing single-modal and simple image-text splicing methods.

[0017] 3. This invention fills the gap in the field of multimodal hazard identification in hydropower engineering. Through the design of multimodal fusion, industry customization and high-performance model, it provides an intelligent solution that is both scientific and practical for the identification of safety hazards in hydropower engineering construction. Its core gating fusion mechanism and classification system are irreplaceable, reducing the reliance on expert experience and adapting to the real-time safety monitoring needs of long-term and large-scale construction in hydropower projects. Attached Figure Description

[0018] The present invention will be further described below with reference to the accompanying drawings and embodiments: Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a comparison chart of losses in multimodal models; Figure 3 To compare the effectiveness of the control model in identifying hazard types; Figure 4 This is a comparison chart showing the differences in indicators for hazard identification between the control and control models. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention.

[0020] Example 1: A multimodal intelligent identification method for safety hazards in hydropower engineering construction, such as... Figure 1 As shown, it includes the following steps: Step 1: Data Input: Collect textual and image data of safety hazards in hydropower construction, and establish a correspondence between hazard texts and hazard images.

[0021] After collecting text and image data of safety hazards in hydropower construction, the dataset was preprocessed by removing text with multiple labels and invalid images. This included setting labels and classifying the data. Hazard labels were set according to the standards and specifications of the "Classification Standard for Enterprise Employee Injury and Death Accidents" (GB6441-86) and the "Guidelines for Safety Management of Water Conservancy and Hydropower Engineering Construction" (SL721-2015). The hazards were classified into 12 categories, including explosion, vehicle injury, electric shock, fall from height, fire, mechanical injury, and object strike. A correspondence between hazard text and hazard image was established.

[0022] Twelve classification features for construction safety hazards were defined, and a correspondence was established between hazard text and hazard images. The 12 defined classifications and features of construction safety hazards are shown in Table 1 below.

[0023]

[0024] Step two, feature extraction: use BERT model to process hidden danger text, extract text features, use multi-head self-attention mechanism Vision Transformer model to process hidden danger image data, and extract image features.

[0025] In order to extract the text features of the safety hidden danger of hydropower engineering construction, the BERT model is used for fine tuning as the text encoder. Unlike traditional one-way language models, the encoding mechanism of BERT can obtain context information from both left and right directions at the same time, so as to capture more comprehensive semantic dependency relationships. The overall framework of BERT model mainly includes two stages: pre-training and fine tuning. In the pre-training stage, BERT uses large-scale unlabeled corpus, and trains by means of self-supervised learning method, the main tasks include mask language model and next sentence prediction. In the fine tuning stage, BERT will use the pre-trained parameters as the initialization weight of the downstream task, and combine the labeled task data to optimize and adjust the whole model parameters end to end, so as to adapt to the specific task target.

[0026] The processed hidden danger text is input into the BERT model to obtain a vector representation containing 768 dimensions, the realization formula is: (1); (2); Among them, w n represents the word in the hidden danger text description statement; t i represents the initial sequence of hidden danger text; represents the hyperparameter of BERT model; represents the vector space; is the vector representation output after the hidden danger text is encoded by BERT model; is the encoding operation process of BERT model, which is a function of mapping text sequence to high-dimensional vector.

[0027] In order to extract the image features of the safety hidden trouble of hydropower engineering construction, the problem of converting image recognition into analogy natural language sequence modeling is considered, and Vision Transformer (ViT) is used as the image encoder of the model. The model structure of Vision Transformer does not depend on traditional convolution operation, but is completely based on multi-head self-attention mechanism in Transformer to model the relationship between image blocks. Unlike convolution operation in CNN, Transformer network is better at processing global image information through self-attention mechanism, which has obvious advantages for identifying multiple types of hidden trouble distribution images in complex construction background.

[0028] In Vision Transformer, patch is a small size image block obtained by uniformly cutting the original hidden trouble image, for example, a large image is divided into multiple 16x16 pixel square blocks; token is the vector representation of image patch or special mark, which is the basic unit of Transformer processing, containing local features or classification marks.

[0029] The specific operation is as follows: The image is cut into small patch blocks as the input of the model. Before the input is delivered to the encoder layer, a token for classification is added, and a token vector for identifying classification is inserted at the beginning of each sequence. For each input token, three vectors , and are generated after input cross feature fusion, and the self-attention and input sequence calculation formulas are as follows: (3) (4) Among them, the function of soft max(·) is to normalize, represents the scaling of the feature; represents the transpose of the vector ; represents the self-attention calculation function generated by converting the patch of hidden trouble image into token sequence; is the vector representation of hidden trouble image after encoding by Vision Transformer model; represents the multi-head self-attention encoding process; represents the connection of multiple vectors in the feature dimension; represents an independent self-attention unit.

[0030] Step three, feature fusion: the extracted text features and image features are fused through the gate recurrent network Gate Net.

[0031] The operation is as follows: The text-based hidden danger modal representation is f text And the image-based hidden danger model is f img As input, the text and image output gates of the gate recurrent network Gate Net are obtained, and the calculation formula is as follows: (5); (6); Wherein, is the output gate of the text, is the output gate of the image.

[0032] Step four, feature processing: the fused features are further processed through the Transformer model to extract higher-level features.

[0033] The gate recurrent network Gate Net is used to fuse the feature expression, and the local feature sequence containing the hidden danger of the image and text information association is obtained. The Transformer model is used to further capture the global dependence, and the deep semantic and logic of the whole image and text are obtained. The features containing higher-level semantics are extracted.

[0034] Step five, hidden danger classification: using a multilayer perceptron (MLP) to classify the processed features, output the results of each type of hidden danger.

[0035] Specifically, the following steps are included: S501, unlike the end-to-end process of the Transformer model which constructs global features based on self-attention, the gate recurrent network Gate Net is used to focus on solving the interaction and screening of image and text multi-modal information, so that the information between different modalities is fully interacted, and the redundant and irrelevant information is effectively eliminated through the gating mechanism. The fused features are then input to the multilayer perceptron for nonlinear mapping and discrimination, and finally the class probability distribution is output: (7); (8); Wherein, x denotes the multi-modal feature vector to be processed in the gate recurrent network; denotes the activation function, which maps the input to the [0, 1] interval, and the output is used as the gating weight (controls xThe transmission ratio of the middle information is realized to realize the gating logic of screening redundant information. represents a multi-layer perception (composed of multiple fully connected layers) that performs a nonlinear feature transformation on the input x .

[0036] S502, using cross-entropy loss function to supervise learning of formula (8), optimizing objective function J is expressed as: (9). wherein, k is the total number of categories, y t is the true label of the first t sample, if the sample belongs to category t , then y t =1, otherwise 0, t is the "category number", t , used to mark "the true category of the sample belongs to" and "traverse all hidden categories", here the label of the sample is associated through "category t"; P t is the corresponding predicted probability value.

[0037] The results of each type of hidden danger include multiple hidden danger types, including falling from a high place, object impact, mechanical injury, electric shock accident, traffic accident, explosion accident, lifting injury, fire accident, collapse accident, civilized construction, equipment damage, and other accidents. 12 typical hidden danger categories.

[0038] Embodiment 2: In this embodiment, the implementation process is illustrated by specific data.

[0039] The historical hidden danger list of the hydropower project is the basic data source for the hidden danger identification of the embodiment. In this paper, 2282 historical hidden danger data of a large hydropower project from 2020 to 2024 are collected, and the division ratio of the training set, the validation set and the test set of the experiment is 7:2:1. In order to ensure the generalization ability of the safety hidden danger multi-modal intelligent identification model for training, 1000 normal pictures in non-engineering field are also collected and arranged. The number of hidden danger types determines the recognition accuracy of the trained model to a certain extent, and the distribution of hidden danger types is uneven, as shown in Table 2, which is an unbalanced data set, which is consistent with the characteristics of the hydropower construction site.

[0040]

[0041] In this experiment, the pre-trained BERT model is used as the text encoder, and the ViT is used as the image encoder. The model (BERT, ViT) is set to batch size=16, learning rate 1x10-5 , training 20 epochs, maximum sequence length 100, experiments are based on PyTorch and NVIDIA A6000 GPU for training and inference, and other optimization strategies are consistent with other mainstream models. In order to compare with the safety hazard multi-modal recognition model (Gate_image_text) proposed in the present application and verify the effectiveness of the model, the following comparison models are set, and their main performance and characteristics are as shown in Table 3.

[0042]

[0043] The comparison and analysis of the recognition effect of the benchmark model are as follows: The recognition effect of different models on construction safety hazards is obviously different in the performance of single text (text), single image (image), image-text combination (image_text) and the model (gate_image_text) of the present application. To some extent, the image-text combination can effectively improve the performance of the construction safety hazard recognition model. Figure 2 As can be seen, the loss value of the training set in the multi-modal joint control group network has a more obvious downward trend than the image branch and the text branch control group. As can be seen from Table 4, the accuracy of the model of the present application is improved by 1.73% compared with the pure text model, and by 12.24% compared with the pure image model; in the recall rate index, it is improved by 23.26% compared with the pure image model; the F1 score is improved by 12.84% compared with the pure image model, and is basically flat compared with the image-text combination model, only improved by 0.78%.

[0044] The image-text combination is better than the pure text or image model, and the model of the present application has the best recognition accuracy, followed by the image-text combination model. The accuracy of the pure text model (84.49%) is slightly higher than that of the image model (76.58%), which shows that the processing of text information is more accurate than that of image information on this data set, because there are a large number of personnel, materials and mechanical equipment in the complex water and electricity engineering construction site, and the image data collected on site often contains a large amount of complex background information in the horizontal and vertical planes, which makes the model prone to interference when extracting hazard features. However, the precision of the model of the present application is the highest, reaching 87.92%, which further shows that the model is more accurate in predicting positive classes and less misidentification. Therefore, the improvement of the multi-modal model in the image hazard identification related task (generally more than 20%) is significantly higher than that of the text task model (1%-4%), thereby verifying the effectiveness of the complementary features of image and text, and the optimization of the image hazard feature by the gate fusion network is particularly prominent, achieving a maximum improvement of 23.26% in the recall (Recall) index of the single-modal hazard image model.

[0045]

[0046] The performance evaluation results of the method of the present application in identifying the types of safety hidden dangers in hydropower engineering construction are shown in Table 5. For “fire accidents” and “explosion accidents”, the precision and recall of the model are close to 1.0, and the F1 score is high, indicating that the model has high accuracy and high recall in identifying such hidden dangers; the identification of “mechanical injury”, “electric shock accident” and “lifting injury” also performs well, with high precision, recall and F1 score. This is mainly due to the typical visual features (such as open flames, smoke, electricity) and clear text description paradigm (such as the terms “wiring” and “flammable materials”) of such hidden dangers, which enable precise alignment of multi-modal features. The identification of “collapse accidents”, “object strikes” and “other accidents” is less effective, mainly due to the ambiguity of visual features (such as the difficulty of capturing subtle cracks) and the use of non-standardized expressions such as “structural abnormalities” in text records, making it difficult for the model to establish an effective feature correlation mechanism. This performance difference essentially reflects the technical challenge of identifying safety hidden dangers in hydropower engineering construction, which is that “explicit features are easy to model, and implicit risks are difficult to capture”.

[0047]

[0048] The F1-score, Recall and Precision performance indicators of the control model on the test set for different types of accidents are shown in Table 6. Figure 3 The present application has more advantages in stability and identification effect. In electric shock accidents and fire accidents, the multi-modal model based on the gating fusion network has consistently high identification performance, even if the data volume percentage fluctuates, it is not significantly affected, and the stability is better than other models. In the identification of mechanical injury hidden dangers, gate_image_text has not reached the highest value, but it is still significantly higher than image_avg, showing its advantage in complex accident data processing; in the identification of hidden dangers such as civilized construction and equipment damage, which are more difficult to identify, the decline in the identification performance of the gate_image_text model is relatively small, and the recovery ability is also stronger.

[0049] The performance of different types of accidents in different indicators is different, which helps to evaluate the performance and accuracy of the model in identifying different types of accidents, as shown in Table 7. Figure 4The model performs well on specific categories and tasks, but still lacks in universality and stability. From the four charts, the relationship between precision and F1 score of different categories on text and image data can be seen. The gating fusion network effectively improves the recognition effect of the multi-modal recognition model on key categories. The precision of most categories is high, especially for the "object strike" and "explosion" categories, which have higher performance than other control models. Therefore, the model with its gating fusion network characteristics performs well in hidden danger type recognition, has strong stability, and has more advantages in recognition effect. Compared with traditional single modal, the multi-modal recognition model can better cope with the data characteristics of complex hidden danger types, and provides a more reliable solution for construction safety hidden danger recognition.

[0050] Embodiment 3: The embodiment provides a water and electricity engineering construction safety hidden danger image-text multi-modal intelligent recognition system, the recognition method in the above embodiment, comprising: The data input module is used for collecting water and electricity engineering construction safety hidden danger text and image data, and establishing a corresponding relationship between hidden danger text and hidden danger image.

[0051] The feature extraction module processes the hidden danger text using a BERT model to extract text features, and processes the hidden danger image data using a multi-head self-attention mechanism Vision Transformer model to extract image features.

[0052] The feature extraction module includes a fine-tuned BERT model text feature extractor and a Vision Transformer model image feature extractor.

[0053] The feature fusion module fuses the extracted text features and image features through a gating recurrent network Gate Net to obtain a fusion feature expression.

[0054] The feature processing module further processes the fused features through a Transformer model to extract higher-level features.

[0055] The hidden danger classification module classifies the processed features using a multi-layer perceptron to output the results of each type of hidden danger.

[0056] Construction process: (1) Build a hardware platform: equip relevant servers; (2) Install the software environment: deploy the PyTorch deep learning framework and the HuggingFace model library; (3) Load the pre-trained model: load the fine-tuned BERT model for the text module and load the VisionTransformer model pre-training parameters for the image module.

[0057] (4) configuring a gating network (5) training the classifier using a cross-entropy loss function Working process: In the running process of the model, the field hidden danger image is collected in real time by a monitoring device and input into the system by an inspection personnel; the image data is input into a Vision Transformer model to divide the image into a sequence of image blocks and generate a visual feature vector, and the text data is input into a fine-tuned BERT model to extract semantic features; the dual-modal features are adaptively weighted and fused through a gating network to eliminate feature redundancy; after the fused features are enhanced by a Transformer layer to strengthen the context association, a multi-layer perceptron (MLP) outputs a hidden danger type probability; when a high-risk hidden danger (such as falling from a height) is identified, the system automatically triggers an alarm and pushes a disposal scheme to a management terminal.

[0058] The embodiment also provides a computer readable storage medium, which stores computer instructions for causing the computer to execute the water conservancy and hydropower engineering construction safety hidden danger image-text multi-modal intelligent identification method described in the above embodiment.

[0059] The embodiment also provides an electronic device, which comprises a memory and a processor, and the memory and the processor are in communication connection with each other; the memory stores computer instructions; and the processor executes the computer instructions to execute the water conservancy and hydropower engineering construction safety hidden danger image-text multi-modal intelligent identification method described in the above embodiment.

[0060] Those skilled in the art can easily understand that the above description is only the preferred embodiment of the present application, and is not intended to limit the present application, and any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A water and electricity engineering construction safety hidden danger graphic and text multi-modal intelligent identification method, characterized in that, The method comprises the following steps: Step one, data input: collect hydropower engineering construction safety hazard text and image data, and establish a corresponding relationship between the hazard text and the hazard image; Step two, feature extraction: process the hazard text using a BERT model to extract text features, and process the hazard image data using a multi-head self-attention mechanism Vision Transformer model to extract image features; Step three, feature fusion: fuse the extracted text features and image features through a gate recurrent network Gate Net to obtain a fused feature expression; Step four, feature processing: further process the fused features through a Transformer model to extract higher-level features; Step five, hazard classification: classify the processed features using a multilayer perceptron to output the results of each type of hazard.

2. The water conservancy project construction safety hidden danger graphic and text multi-modal intelligent identification method according to claim 1, characterized in that, In step one, after collecting the hydropower engineering construction safety hazard text and image data, the text multi-label and invalid images are removed, the data set is preprocessed, including setting labels, dividing, and establishing a corresponding relationship between the hazard text and the hazard image.

3. The water conservancy project construction safety hidden danger image and text multi-modal intelligent identification method according to claim 1, characterized in that, In step two, the processed hazard text is input into the BERT model to obtain a 768-dimensional vector representation, and the formula is: （1）； （2）； wherein, w n representing a word in a hazard textual description sentence; t i representing an initial sequence of hazard text; representing hyperparameters of a BERT model; representing a vector space; is a vector representation of the hazard text output after encoding by the BERT model; is an encoding operation process of the BERT model, which is a function of mapping a text sequence to a high-dimensional vector.

4. The water conservancy construction safety hidden danger image and text multi-modal intelligent identification method according to claim 1, characterized in that, In step two, the hazard image data is processed using a multi-head self-attention mechanism Vision Transformer model to extract image features. In the multi-head self-attention mechanism Vision Transformer model, patch is a small size image block obtained by uniformly cutting the original hazard image, token is a vector representation of the image patch or a special marker, and is the basic unit of Transformer processing; The specific operation of step two is as follows: Cut the image into patches as the input of the model, add a token for classification before the input is passed to the encoder layer, insert a token vector for identifying classification at the beginning of each sequence, and generate three vectors for each input token after cross-feature fusion 、 and , and the calculation formulas of self-attention and input sequence are as follows: （3）； （4）； where the soft max(·) function is used for normalization, denotes the scaling of the feature; denotes the transpose of the vector ; denotes the patch-to-token conversion of the hazard image to generate the self-attention computation function; denotes the multi-head self-attention encoding process; is the vector representation of the hazard image after encoding by the Vision Transformer model; denotes the concatenation of multiple vectors in the feature dimension; denotes an independent self-attention unit.

5. The method according to claim 1, wherein the method according to claim 1 is characterized by, In step three, the extracted text features and image features are fused through a gate recurrent network Gate Net to obtain a fused feature expression, and the operation is as follows: The text-based hazard modal representation is f text And the image-based hazard modal representation is f img As input, the output gate of the text and image in the gated recurrent network Gate Net is input respectively, and the calculation formula is as follows: （5）； （6）； wherein, is an output gate for text; is an output gate for images.

6. The water conservancy construction safety hidden danger image and text multi-modal intelligent identification method according to claim 1, characterized in that, In step five, the hazard classification specifically comprises the following steps: S501, use the gate recurrent network Gate Net to solve the interaction and screening of the graphic and text multi-modal information, so that the information between different modalities is fully interacted, and the redundant and irrelevant information is eliminated through the gating mechanism, and the fused features are then input into a multilayer perceptron for nonlinear mapping and discrimination, and finally the class probability distribution is output: （7）； （8）； wherein, x denotes a multi-modal feature vector to be processed in the gated recurrent network; denotes an activation function mapping the input to the interval [0, 1] with the output as a gating weight; denotes a multi-layer perceptron performing a non-linear feature transformation on the input x ; S502, adopt cross entropy loss function to supervise the category probability prediction model to learn, optimization objective function J is expressed as: （9）； wherein, k is the total number of classes, y t is the true label of the t th sample, if the sample belongs to class t , t , then y t = 1, otherwise 0; P t is the corresponding predicted probability value.

7. The water conservancy construction safety hidden danger image and text multi-modal intelligent identification method according to claim 1, characterized in that, In step five, the results of each type of hazard include multiple hazard types, including explosion accidents, vehicle injuries, electric shock accidents, falls from a height, fire accidents, mechanical injuries, object strikes, lifting injuries, collapse accidents, equipment damage, civilized accidents, and other accidents, totaling 12 types.

8. A water and electricity engineering construction safety hidden danger image and text multi-modal intelligent identification system for the identification method of any one of claims 1-7, characterized in that, It comprises: A data input module for collecting hydropower engineering construction safety hazard text and image data, and establishing a corresponding relationship between the hazard text and the hazard image; A feature extraction module for processing the hazard text using a BERT model to extract text features, and processing the hazard image data using a multi-head self-attention mechanism Vision Transformer model to extract image features; The feature fusion module fuses the extracted text features and image features through a gate recurrent network Gate Net to obtain a fused feature expression. The feature processing module further processes the fused features through a Transformer model to extract higher-level features. The hazard classification module classifies the processed features using a multilayer perceptron to output results of various hazards.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for causing the computer to execute the water conservancy project construction safety hazard image-text multi-modal intelligent recognition method according to any one of claims 1-7.

10. An electronic device, comprising: Comprise: A memory and a processor, which are in communication connection with each other, the memory stores computer instructions, and the processor executes the computer instructions to execute the water conservancy project construction safety hazard image-text multi-modal intelligent recognition method according to any one of claims 1-7.

Images