Water transportation construction project power distribution box electricity safety monitoring method based on machine vision

Abstract

The application discloses a kind of water transport construction engineering distribution box electric safety monitoring methods based on machine vision, comprising: A1: collection distribution box high-definition image data, the text data of distribution box electric safety standard, and separately pre-process;A2: extract multi-scale fusion text features, domain enhanced text features, text semantic entity set;A3: extract global image features, calculate enhanced image features, entity category probability matrix, entity position coordinate matrix, then extract image semantic entity matrix;A4: according to text semantic entity set, image semantic entity matrix, calculate missing image semantic set;A5: extract image key area mask;A6: according to image key area mask, enhanced image features, domain enhanced text features, missing image semantic set, output safety hazard analysis result;The application can solve the problem that traditional method is difficult to associate distribution box safety hazard identification result with standard specification.

Description

Technical Field

[0001] This invention relates to the field of safety in waterway construction projects, and in particular to a method for monitoring the electrical safety of distribution boxes in waterway construction projects based on machine vision. Background Technology

[0002] At waterway construction sites, distribution boxes are core equipment for power supply. Their safe and stable operation directly affects the construction progress, the personal safety of construction workers, and the normal operation of surrounding waterway facilities. Therefore, stricter requirements are placed on the electrical safety monitoring of distribution boxes: not only must they be able to accurately identify faulty and missing hazards in the distribution boxes, but they must also be able to effectively link the identified hazards with the corresponding safety regulations, providing clear standard basis for hazard rectification and law enforcement verification. However, this scenario itself has many constraints, mainly reflected in two core aspects: regulatory texts and on-site images.

[0003] On the one hand, at the level of the standard text, the safety standard text for distribution boxes in waterway engineering contains various types of key information, including strong constraint terms and object component terms. The language is complex and contains a large number of industry-specific terms. General natural language processing models are prone to misunderstandings of terminology, leading to semantic parsing biases and making it difficult to accurately extract the core constraint requirements in the standard clauses. This directly affects the accuracy of the association between hidden dangers and the standard. On the other hand, at the level of on-site images, the construction site environment of waterway engineering is complex. The images of distribution boxes taken generally have problems such as cluttered backgrounds. Moreover, the hidden dangers are mostly structural problems involving component relationships, locations, or overall states, rather than single-point texture problems, which further increases the difficulty of accurately associating hidden dangers with standard clauses.

[0004] Traditional methods for monitoring electrical safety in distribution boxes primarily rely on professional inspectors periodically conducting manual inspections of distribution boxes at construction sites. This involves visually observing the box's appearance, wiring, and labeling integrity, combined with measurements of electrical parameters using tools such as multimeters, to determine the presence of potential electrical safety hazards. This method is not only costly in terms of manpower, resources, and time, but also suffers from low monitoring efficiency, subjective hazard identification, easy omission of hidden hazards, inability to quickly link to regulatory provisions, and poor guidance for rectification, making it difficult to meet the large-scale, 24 / 7 safety monitoring needs of waterway construction projects. Furthermore, it is ill-suited for the precise identification of similar components and missing parts in complex scenarios.

[0005] With the rapid development of deep learning technology, machine vision-based monitoring methods are gradually being applied to the field of electrical safety monitoring of distribution boxes. The general process usually involves acquiring image data of distribution boxes at the construction site of waterway construction projects through image acquisition equipment, and using deep learning models such as convolutional neural networks to extract and identify features from the image data in order to determine whether there are any potential electrical safety hazards in the distribution boxes. However, existing technologies still have many prominent pain points: In terms of standard text processing, existing technologies mostly adopt a single-granularity text encoding method, only performing simple semantic extraction on the text, without combining it with a terminology dictionary for distribution boxes for targeted terminology enhancement processing. This makes it impossible to accurately parse the multi-type semantic information of standard texts, and easily leads to problems such as inaccurate reading of clauses and misunderstanding of terminology. In terms of visual monitoring, existing deep learning-based machine vision monitoring methods mostly focus on extracting local texture features of images, lacking global modeling of the overall structure of the box and the positional relationships between components, and cannot support the identification of structural hazards. In terms of cross-modal correlation, existing technologies can mostly only process text or image information separately, resulting in cross-modal semantic gaps. They cannot effectively align the parsed standard clauses with the on-site image monitoring results, making it difficult to identify the specific standard clauses violated by the hazards. At the same time, it is difficult to integrate hazard information and standard clauses into standard conclusions that can be directly issued, resulting in insufficient guidance of monitoring results. Managers cannot quickly locate the root cause of hazards based on monitoring results, and rectification personnel cannot accurately implement rectification based on monitoring results. Summary of the Invention

[0006] In view of this, the present invention aims to provide a method for monitoring the electrical safety of distribution boxes in waterway construction projects based on machine vision, so as to solve the problems of insufficient accuracy in hazard identification of traditional deep learning monitoring methods and difficulty in effectively associating hazard identification results with relevant standards and specifications.

[0007] A machine vision-based method for monitoring the electrical safety of distribution boxes in waterway construction projects includes: A1: Collect high-definition image data of the distribution box, text data of the electrical safety standards for the distribution box, and terminology data in the field of distribution boxes, and preprocess them separately to obtain preprocessed image data of the distribution box, preprocessed text data of the safety standards for the distribution box, and a dictionary of terminology in the field of distribution boxes. A2: Based on the preprocessed safety standard and specification text data of the distribution box, multi-scale fused text features are extracted, and then combined with the terminology dictionary of the distribution box domain, domain-enhanced text features are extracted, and then the set of text semantic entities is extracted. A3: Based on the preprocessed distribution box image data, extract global image features, calculate enhanced image features, entity category probability matrix, entity location coordinate matrix, and then extract the image semantic entity matrix; A4: Calculate the aligned text enhancement features and the missing image semantic set based on the text semantic entity set and the image semantic entity matrix; A5: Extract the mask of key regions in the image based on the enhanced image features and the aligned text enhancement features; A6: Based on the image key area mask, enhanced image features, domain-enhanced text features, and missing image semantic set, output the security risk analysis result text.

[0008] Furthermore, step A1 also includes: A11: High-definition image data of power distribution boxes in waterway construction scenes are acquired by high-definition industrial cameras, and Gaussian filtering and scale normalization are performed to obtain pre-processed power distribution box image data. A12: Collect text data of electrical safety standards for distribution boxes in waterway construction scenarios. The data type is unstructured text data. Then, perform text cleaning and jieba word segmentation to obtain preprocessed text data of electrical safety standards for distribution boxes. A13: Collect terminology data in the field of distribution boxes, perform text cleaning processing, and store it in dictionary format to obtain a terminology dictionary for the field of distribution boxes.

[0009] Furthermore, step A2 also includes: A21: Based on the preprocessed safety standard and specification text data for distribution boxes, character-level text encoding features, word-level text encoding features, and sentence-level text encoding features are extracted using convolutional layers, bidirectional long short-term memory networks, and Transformer encoders. The calculation method is as follows: ; ; ; ; in, These are the character-level embedding vectors, word-level embedding vectors, and sentence-level embedding vectors of the preprocessed distribution box safety standard specification text data. To pre-train a bidirectional large language model, This is the pre-processed text data of the safety standard specifications for the distribution box. Character-level text encoding features, It is a one-dimensional convolutional layer with residual connections. For word-level text encoding features, It is a bidirectional long short-term memory network. For sentence-level text encoding features, For Transformer encoders; A22: Based on character-level text encoding features, word-level text encoding features, and sentence-level text encoding features, multi-scale fused text features are extracted through a multi-scale semantic collaborative gating fusion mechanism. The calculation method is as follows: ; ; in, For multi-scale gated weight vectors, For the Softmax function, It is a multilayer perceptron. For feature splicing operations, To fuse text features at multiple scales, These are the first, second, and third feature components of the multi-scale gated weight vector, respectively. For Hadama accumulation, This is element-wise addition; A23: Based on multi-scale fused text features and a terminology dictionary for the distribution box domain, domain-enhanced text features are extracted through a terminology attention enhancement mechanism. The calculation method of the terminology attention enhancement mechanism is as follows: ; ; ; ; ; in, This is a coding vector for distribution box terminology. A dictionary of terms for the field of electrical distribution boxes. For the term attention weight matrix, For feature dimension, To match the feature matrix for the terms, For term gain weight, To enhance textual features for the domain, for function; A24: Based on domain-enhanced text features and multi-scale gated weight vectors, the text semantic entity matrix is ​​extracted using a conditional random field decoder. The calculation method is as follows: ; in, For text semantic entity matrix, For conditional random field decoders; A25: Using the Viterbi algorithm, the label sequence with the highest probability value is selected from the text semantic entity matrix to obtain the text semantic entity set. .

[0010] It should be further noted that the safety specifications for distribution boxes typically contain multiple types of key information simultaneously, such as strong constraint terms like "must," "strictly prohibited," "should," and "must not," and terms related to components like "residual current device," "box door," and "nameplate." Furthermore, the language of the specifications is complex; using only a single granularity to represent the clauses can lead to inaccurate reading. For example, some related terms differ significantly at the character level and may be affected by segmentation or synonymous expressions at the word level. In addition, the standard text for distribution boxes in waterway engineering contains a large number of industry-specific terms. For instance, general natural language processing models may easily misunderstand "grounding" as "contact with the ground" rather than "electrical grounding protection," leading to inaccurate semantic understanding. This invention first leverages the multi-type semantic characteristics of standard texts, employing a BERT-based approach combined with various processing methods. Through convolutional layers, bidirectional long short-term memory networks, and Transformer encoders, it extracts character-level, word-level, and sentence-level text encoding features. These different granularities of features can be adapted to different types of information within the text: character-level features are suitable for capturing abbreviations, numbers, numerical values, and other detailed information; word-level features are suitable for capturing terminology collocations and local semantics; and sentence-level features are suitable for capturing the constraint logic and conditional structure of entire sentences, thus achieving a comprehensive understanding of standard texts. Based on this, a multi-scale semantic collaborative gating fusion mechanism is designed. By generating a gating weight vector, the weights of features at different granularities are dynamically allocated, avoiding the dilution of key points in clauses caused by simple splicing or average fusion, resulting in multi-scale fused text features. Subsequently, a terminology attention enhancement mechanism was designed, which combines a terminology dictionary for the distribution box field to perform attention matching and inject terminology gain, thereby solving the problem of insufficient terminology targeting caused by synonyms and abbreviations in the standard text, and accurately mapping the objects, actions, and constraints in the clauses to industry standard terms; finally, the semantic entity matrix of the text was extracted by combining a conditional random field decoder with a gated weight vector, and the Viterbi algorithm was used to select the label sequence with the highest probability. Existing technologies typically employ a single-granularity text encoding method when processing safety regulations for distribution boxes, performing only simple semantic extraction without incorporating a terminology dictionary specific to the distribution box field for targeted terminology enhancement. In contrast, this invention offers the following advantages: it solves the problem of inaccurate reading of regulation texts through multi-granularity text encoding; it addresses the dilution of key clauses through gating fusion; and it addresses the lack of terminology specificity through terminology attention enhancement. These improvements enhance the efficiency and accuracy of regulation text processing, better aligning with the actual needs of distribution box enforcement and rectification.

[0011] Furthermore, step A3 also includes: A31: Input the preprocessed distribution box image data into the Transformer encoder, construct a feature sequence through convolutional mapping and positional encoding, and then use the Transformer encoder to extract global image features. The calculation method is as follows: ; in, For global image features, For flattening operation, It is a convolutional layer. The image data of the preprocessed distribution box. This is the position encoding matrix; A32: Based on global image features, feature enhancement is performed through a multi-scale feature fusion mechanism to obtain enhanced image features. The calculation method is as follows: ; ; in, Image mean features, For mean pooling operation, It is a vector of all 1s. These are the global image feature weight matrix and the image mean feature weight matrix, respectively. For enhanced image features, For layer normalization operation; A33: Based on the enhanced image features, the entity category probability matrix and entity location coordinate matrix are calculated using the classification prediction head and regression prediction head, respectively. Then, the image semantic entity matrix is ​​extracted. The calculation method is as follows: ; ; ; in, This is the entity category probability matrix. This is the category probability weight matrix. This is the class probability bias vector. This is the entity's position coordinate matrix. This is the position coordinate weight matrix. The position coordinate offset vector, For image semantic entity matrix, For indicator functions, To obtain the maximum value, The confidence threshold for semantic entities in an image.

[0012] It should be further explained that, in images, most of the hidden dangers of distribution boxes are not single-point texture problems, but rather structural issues involving the relationships, positions, or overall state of components. For example, whether the markings are in a visible position, whether the cables enter from the correct inlet, etc. At the same time, the shooting conditions at construction sites are complex, with common problems such as cluttered backgrounds, partial obstruction, and shooting angle deviations, which can easily interfere with the detection. In addition, subsequent alignment of the clause entities with the image entities is required, and the image output must contain entity-level information including category and location. Otherwise, it is impossible to point out the evidence area required for rectification, or to determine whether the components required by the clause are missing. When processing images of distribution boxes at the scene, existing technologies usually use traditional image detection methods, which focus on extracting local texture features of the image, lacking global modeling of the overall structure of the box and the positional relationships between components, and the output results are mostly single component detection signals, rather than entity-level information including category and location. Addressing the scenario where potential hazards in electrical distribution boxes are primarily structural and systemic issues, this invention first performs convolutional mapping and positional encoding on the preprocessed distribution box image data. Then, a Transformer encoder extracts global image features. Convolutional mapping captures local image details, while positional encoding preserves component positional relationships. Combined with the global modeling capabilities of the Transformer encoder, this effectively supports structural compliance checks, enabling the assessment of the overall box status and inter-component relationships, resulting in global image features. To address local interference issues caused by complex on-site shooting conditions, this invention employs a multi-scale feature fusion mechanism for feature enhancement, extracting global image features... The mean features of the image are used to construct the overall context, which is then fused with global image features and subjected to layer normalization. This injects the overall context of the container into the features, reducing the misleading effect of local salient but irrelevant regions and ensuring that the detection results fit the overall state of the container, resulting in enhanced image features. For the business requirement of aligning subsequent clauses with image entities, this invention uses a classification prediction head and a regression prediction head to calculate the entity category probability matrix and the entity location coordinate matrix, respectively. After concatenating the two, they are filtered through a confidence threshold. The output image semantic entity matrix is ​​entity-level information, which clarifies the category and location of compliance-related components in the image, and finally realizes the transformation of the on-site image into an entity matrix that can be aligned with the clauses.

[0013] Furthermore, step A4 also includes: A41: Based on the text semantic entity set and the image semantic entity matrix, cross-modal semantic alignment is performed through an attention mechanism to obtain aligned text enhancement features; A42: Output the missing image semantic set based on the aligned text enhancement features.

[0014] Furthermore, the calculation process for the missing image semantic set in step A4 includes: First, the text semantic entity set is processed by BERT. The processing result, along with the image semantic entity matrix, is input into an attention mechanism for cross-modal semantic alignment to obtain aligned text augmentation features. Then, the BERT processing result of the text semantic entity set is concatenated with the aligned text augmentation features, and the concatenated result is processed by a multilayer perceptron and input into a sigmoid function to obtain an entity existence score vector. Finally, the existence score of each text entity in the entity existence score vector is compared with a text entity existence score threshold, and text semantic entities with existence scores below the threshold in the text semantic entity set are selected to obtain the missing image semantic set.

[0015] It should be further explained that even if the normative clauses are extracted into text semantic entity sets and the on-site images are detected into image semantic entity matrices, the two still cannot achieve an effective correspondence. For example, the components detected in the images are difficult to match with the components required in the clauses, and mismatch problems are prone to occur when multiple entities appear at the same time. At the same time, there are a large number of hidden dangers of "missing components" in on-site law enforcement rectification. These hidden dangers cannot be explained by image classification alone, nor can they be matched with each clause of the normative clause. This invention addresses the pain point of cross-modal semantic fragmentation by designing a technical solution for cross-modal semantic alignment using an attention mechanism. This mechanism uses the encoded features of a set of text semantic entities as the query basis and the image semantic entity matrix as the matching object. Leveraging the dynamic weight allocation principle of the attention mechanism, the semantic similarity between text semantic entities and each image semantic entity is calculated. Image entities with high semantic matching scores to text entities are assigned higher weights, while semantically irrelevant image entities are filtered out, thus achieving precise matching. For example, the text entity corresponding to "the box door should be locked" will be quickly captured by this mechanism, and the image entity corresponding to "the door lock" will be assigned a high weight, while other component features in the image unrelated to the text entity will be ignored. Entities in the text semantic entity set are matched to the corresponding visual evidence in the image semantic entity matrix, achieving effective association between clause entities and image entities, thereby obtaining aligned text enhancement features. To address the pain point of difficulty in identifying missing items and their inability to correspond to relevant clauses, this invention outputs missing items based on aligned text-enhanced features: The encoded features of the text semantic entity set are concatenated with the aligned text-enhanced features. Then, an entity existence score vector is calculated using a multilayer perceptron and activation function. Finally, the missing image semantic sets are filtered out based on the score threshold. The design of concatenating two features is because the encoded features of the text semantic entity set clearly define "entities that the clause requires to exist," while the aligned text-enhanced features reflect "whether the corresponding image entity exists." Combining the two allows the model to simultaneously grasp the inspection standards and the actual situation on-site. The multilayer perceptron is used to mine the correlation between the two features, transforming this correlation into a quantifiable entity existence score. The activation function then normalizes the score within a reasonable range, facilitating a clear determination of entity existence. The score threshold design avoids fuzzy judgments, ensuring that only when the score is below the threshold—that is, when there is no effective correlation between the text entity and the image entity—is it considered missing. For example, if the score of the text entity corresponding to the "warning sign" required by the clause is below the threshold after feature concatenation, it means that the corresponding warning sign was not detected in the image, thus determining it as a missing item. Existing technologies can mostly only process text or image information individually, resulting in cross-modal semantic gaps, making it difficult to effectively identify potential missing information and to match missing items with each clause of the specification. Compared with existing technologies, the advantages of this invention are: This invention achieves cross-modal semantic alignment between text and images through an attention mechanism, effectively solving the problem of semantic gaps, ensuring that clause requirements correspond to on-site image evidence, and then outputting a set of missing image semantics through entity existence score calculation and threshold filtering, achieving a one-to-one correspondence between missing items and clauses, which is more in line with the needs of actual business scenarios.

[0016] Furthermore, the process of extracting the image key region mask in step A5 includes: First, the aligned text enhancement features and the enhanced image features are multiplied by an inner product. The result is divided by the square root of the feature dimension and then processed by the Softmax function to obtain the entity-guided spatial attention matrix. Next, a spatial dimension broadcasting operation is performed on the entity-guided spatial attention matrix. The broadcasting result is then multiplied by the enhanced image features using a Hadamard product, and processed by a 3×3 convolutional layer to obtain attention-weighted guided features. Finally, the attention-weighted guided features are batch-normalized, and the result is input into the Sigmoid function. The difference between the output of the Sigmoid function and the image key region mask threshold is calculated, and the difference is input into a step function to obtain the image key region mask.

[0017] It should be further explained that there are many internal components in the distribution box, and many of these components have similar shapes and structures. If the key areas are not focused on first and error identification is carried out directly in the entire image, firstly, the system processing complexity will be greatly increased, as all similar components in the entire image need to be identified and judged one by one; secondly, similar components in non-key areas may be misjudged as incorrect parts, resulting in false error prompts. To address this pain point, this invention designs an image key region mask. An entity-guided spatial attention matrix is ​​calculated using aligned text enhancement features and enhanced image features. Then, attention weighting, convolution, batch normalization, thresholding, and binarization are applied to the enhanced image features to finally obtain the key region mask. The spatial attention matrix, guided by the aligned text enhancement features, first identifies key regions relevant to the requirements, focusing the recognition range within these regions and eliminating interference from numerous similar components outside these regions—reducing system processing complexity, minimizing unnecessary computation, and avoiding invalid identification of similar components in non-key regions. After focusing on the key regions, error identification is performed within these regions, distinguishing subtle differences between components and avoiding misjudgments caused by similar components outside the regions. This ensures that truly erroneous parts within the key regions are identified. For example, if a requirement is to inspect the grounding terminal in a distribution box, the system first focuses on the key region where the grounding terminal is located using the key region mask, eliminating interference from other similar terminals, and then identifies whether there are wiring errors in the grounding terminal within this region. This improves recognition efficiency and ensures recognition accuracy.

[0018] Furthermore, step A6 also includes: A61: Extract the image focus features and compliance difference features of key regions based on the image key region mask, enhanced image features, and domain-enhanced text features; A62: Based on compliance difference characteristics, key area image focus characteristics, and missing image semantic sets, output the security risk analysis result text.

[0019] Furthermore, step A6 also includes: First, perform a Hadamard product between the enhanced image features and the image key region mask, and then perform global average pooling on the product result to obtain the image focus features of the key region. Next, calculate the inner product of the key region image focus feature and the aligned text enhancement feature. Divide the inner product result by the square of the vector norm of the aligned text enhancement feature. Then multiply the calculated result with the aligned text enhancement feature. Finally, subtract the multiplication result from the key region image focus feature to obtain the orthogonal projection difference vector. Next, the difference between the orthogonal projection difference vector and the focus feature of the key region image minus the aligned text enhancement feature is concatenated. The concatenated result is then processed by a multilayer perceptron and input into the hyperbolic tangent function to obtain the compliance difference feature. The compliance difference features are then processed through a fully connected layer to obtain the security risk confidence probability. The security risk confidence probability is then concatenated with the compliance difference features and input into the large language model decoder to obtain the preliminary security risk analysis result text. Finally, the missing image semantic set and the preliminary security risk analysis result text are input into the large language model for processing to obtain the security risk analysis result text.

[0020] It should be further explained that in the scenario of power distribution box safety monitoring, it is necessary not only to identify violations and errors in key areas, but also to identify missing defects and output safety hazard analysis results that can be directly issued and are easy to understand and standardized. Most existing technologies can only identify single error defects, cannot take into account both at the same time, and are difficult to associate with standards and specifications to integrate relevant information into standardized conclusions that can be directly issued. Based on the key area features and cross-modal alignment features discussed above, this invention can identify violations in key areas, such as incorrect grounding terminal wiring, as well as defects such as missing warning signs. Finally, the confidence level of the hidden danger is confirmed through a fully connected layer, and the large language model decoder initially generates the hidden danger conclusion. The missing items are then integrated for secondary consolidation, and finally, a complete hidden danger analysis result is output.

[0021] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) This invention effectively solves the problems of insufficient accuracy in hazard identification, cross-modal semantic gaps, and inability to effectively associate hazards with safety regulations in traditional monitoring methods. First, high-definition image data of distribution boxes and safety regulations text data are preprocessed. Then, relevant features and entity information of text and images are extracted respectively. By association and alignment, the modal barriers between text and images are broken down, key areas are focused on and irrelevant interference is reduced. Combined with the missing entity information, the final safety hazard analysis result text that can be implemented in accordance with regulations is generated. This invention significantly reduces the manpower and material costs of monitoring, significantly improves the accuracy and efficiency of hazard identification, and realizes the effective association between hazards and safety regulations. It provides clear standard basis for monitoring results and provides clear guidance for hazard rectification and law enforcement verification.

[0022] (2) In view of the fact that the safety specifications for distribution boxes are characterized by a variety of text information, complex language and a large number of industry-specific terms, the existing single-granularity coding technology is prone to problems such as inaccurate text reading and insufficient terminology targeting. This solution is based on BERT, combined with multi-module extraction of multi-granularity text coding features, dynamically allocates feature weights through a multi-scale semantic collaborative gating fusion mechanism, and combines a terminology dictionary for the distribution box field to achieve terminology attention enhancement, accurately extracting the set of semantic entities in the text; effectively solving the problems of deviation in specification text parsing and misunderstanding of terminology, improving the efficiency and accuracy of text processing, and accurately extracting the core constraints in the clauses, laying a precise and reliable text foundation for the subsequent association between hidden dangers and specification clauses.

[0023] (3) In view of the fact that most of the hidden dangers of the distribution box are structural problems and the background of the on-site captured images is cluttered and other interference, the existing technology only focuses on local texture features. This invention performs convolution mapping and position encoding on the preprocessed image, extracts global image features through the Transformer encoder, and then achieves feature enhancement through a multi-scale feature fusion mechanism. Combined with the classification and regression prediction head, it outputs an entity-level image semantic entity matrix. It effectively captures the overall state of the distribution box and the positional relationship between the components, reduces misjudgment caused by on-site image interference, and the output entity-level information can clearly identify the category and position of the compliant related components. It realizes the accurate extraction of image features, provides reliable image data support for subsequent cross-modal entity alignment, and improves the targeting and accuracy of image detection.

[0024] (4) In view of the problem that there is a cross-modal semantic gap between text and image entities and that it is difficult to judge the missing items on site, this invention uses an attention mechanism to perform cross-modal semantic alignment between the text semantic entity set and the image semantic entity matrix, and then splices the text semantic entity encoding features and the aligned text enhancement features to calculate the entity existence score and filter the missing image semantic set according to the threshold. This realizes the effective association between clause entities and image entities, and accurately identifies the missing items and matches them with the corresponding standard clauses one by one. This makes up for the shortcomings of the existing technology in not being able to take into account the identification of missing items and the association with the standard, making the identification of the hidden danger more comprehensive and providing a clear basis for the determination of missing items for on-site law enforcement rectification.

[0025] (5) In view of the problem that the large number of internal components of the distribution box and their similar shape and structure make it easy to misjudge similar components in non-key areas when the full image range is directly identified, the present invention constructs an entity-guided spatial attention matrix based on the aligned text enhancement features and the enhanced image features. After a series of processing, an image key area mask is generated, which focuses the error identification range on the key areas related to the clauses. This reduces the processing complexity of the system, reduces unnecessary computation, effectively eliminates the interference of similar components outside the area, avoids the occurrence of false error prompts, ensures that the real error parts in the area are identified, significantly improves the efficiency and accuracy of hidden danger identification, and is suitable for the identification scenario of multiple similar components in the distribution box. Attached Figure Description

[0026] Figure 1 A flowchart illustrating a method for monitoring the electrical safety of distribution boxes in waterway construction projects based on machine vision, provided by the present invention. Figure 2 This is a schematic diagram illustrating the defect identification and standard correlation results for the compliance testing of distribution boxes provided by the present invention. Detailed Implementation

[0027] The present invention will be further described below with reference to the accompanying drawings, but this is not intended to limit the present invention in any way. Any modifications or substitutions made based on the teachings of the present invention shall fall within the protection scope of the present invention.

[0028] Example 1: A method for monitoring the electrical safety of distribution boxes in waterway construction projects based on machine vision, such as... Figure 1 As shown, it includes the following steps: A1: Collect high-definition image data of the distribution box, text data of the electrical safety standards for distribution boxes, and terminology data in the field of distribution boxes. Perform preprocessing on these data to obtain preprocessed image data of the distribution box, preprocessed text data of the electrical safety standards for distribution boxes, and a terminology dictionary for the field of distribution boxes, including: A11: High-definition image data of power distribution boxes in waterway construction scenes are acquired by high-definition industrial cameras, and Gaussian filtering and scale normalization are performed to obtain pre-processed power distribution box image data. A12: Collect text data of electrical safety standards for distribution boxes in waterway construction scenarios. The data type is unstructured text data. Then, perform text cleaning and jieba word segmentation to obtain preprocessed text data of electrical safety standards for distribution boxes. A13: Collect terminology data in the field of distribution boxes, perform text cleaning processing, and store it in dictionary format to obtain a terminology dictionary for the field of distribution boxes.

[0029] A2: Based on the preprocessed safety standard and specification text data for distribution boxes, multi-scale fused text features are extracted. Then, combined with a domain-specific terminology dictionary for distribution boxes, domain-enhanced text features are extracted, leading to the extraction of a set of textual semantic entities, including: A21: Based on the preprocessed safety standard and specification text data for distribution boxes, character-level text encoding features, word-level text encoding features, and sentence-level text encoding features are extracted using convolutional layers, bidirectional long short-term memory networks, and Transformer encoders. The calculation method is as follows: ; ; ; ; in, These are the character-level embedding vectors, word-level embedding vectors, and sentence-level embedding vectors of the preprocessed distribution box safety standard specification text data. To pre-train a bidirectional large language model, This is the pre-processed text data of the safety standard specifications for the distribution box. Character-level text encoding features, It is a one-dimensional convolutional layer with residual connections. For word-level text encoding features, It is a bidirectional long short-term memory network. For sentence-level text encoding features, For Transformer encoders; A22: Based on character-level text encoding features, word-level text encoding features, and sentence-level text encoding features, multi-scale fused text features are extracted through a multi-scale semantic collaborative gating fusion mechanism. The calculation method is as follows: ; ; in, For multi-scale gated weight vectors, For the Softmax function, It is a multilayer perceptron. For feature splicing operations, To fuse text features at multiple scales, These are the first, second, and third feature components of the multi-scale gated weight vector, respectively. For Hadama accumulation, This is element-wise addition; A23: Based on multi-scale fused text features and a terminology dictionary for the distribution box domain, domain-enhanced text features are extracted through a terminology attention enhancement mechanism. The calculation method of the terminology attention enhancement mechanism is as follows: ; ; ; ; ; in, This is a coding vector for distribution box terminology. A dictionary of terms for the field of electrical distribution boxes. For the term attention weight matrix, For feature dimension, To match the feature matrix for the terms, For term gain weight, To enhance textual features for the domain, for function; A24: Based on domain-enhanced text features and multi-scale gated weight vectors, the text semantic entity matrix is ​​extracted using a conditional random field decoder. The calculation method is as follows: ; in, For text semantic entity matrix, For conditional random field decoders; A25: Using the Viterbi algorithm, the label sequence with the highest probability value is selected from the text semantic entity matrix to obtain the text semantic entity set. .

[0030] A3: Based on the preprocessed distribution box image data, extract global image features, calculate the enhanced image features, entity category probability matrix, and entity location coordinate matrix, and then extract the image semantic entity matrix, including: A31: Input the preprocessed distribution box image data into the Transformer encoder, construct a feature sequence through convolutional mapping and positional encoding, and then use the Transformer encoder to extract global image features. The calculation method is as follows: ; in, For global image features, For flattening operation, It is a convolutional layer. The image data of the preprocessed distribution box. This is the position encoding matrix; A32: Based on global image features, feature enhancement is performed through a multi-scale feature fusion mechanism to obtain enhanced image features. The calculation method is as follows: ; ; in, Image mean features, For mean pooling operation, It is a vector of all 1s. These are the global image feature weight matrix and the image mean feature weight matrix, respectively. For enhanced image features, For layer normalization operation; A33: Based on the enhanced image features, the entity category probability matrix and entity location coordinate matrix are calculated using the classification prediction head and regression prediction head, respectively. Then, the image semantic entity matrix is ​​extracted. The calculation method is as follows: ; ; ; in, This is the entity category probability matrix. This is the category probability weight matrix. This is the class probability bias vector. This is the entity's position coordinate matrix. This is the position coordinate weight matrix. The position coordinate offset vector, For image semantic entity matrix, For indicator functions, To obtain the maximum value, The confidence threshold for semantic entities in an image.

[0031] A4: Based on the text semantic entity set and the image semantic entity matrix, calculate the aligned text enhancement features and the missing image semantic set, including: A41: Based on the text semantic entity set and the image semantic entity matrix, cross-modal semantic alignment is performed through an attention mechanism to obtain the aligned text enhancement features. The calculation method is as follows: ; in, Enhance features for aligned text; A42: Based on the aligned text enhancement features, output the missing image semantic set, calculated as follows: ; ; in, This is the entity existence score vector. For the missing image semantic set, Let i be the i-th text semantic entity in the set of text semantic entities. Let i be the existence score of the i-th text entity in the entity existence score vector. This is the threshold for the existence score of text entities.

[0032] Specifically, for scenarios in waterway construction projects where power distribution boxes are densely distributed with multiple entities and have similar component features, making cross-modal semantic mismatches likely, this invention also provides a cross-modal attention calculation method with layered normalization and multilayer perceptron feature enhancement to replace step A41. The calculation method is as follows: .

[0033] A5: Based on the enhanced image features and the aligned text enhancement features, extract the mask for the key regions of the image. The calculation method is as follows: ; ; ; in, Spatial attention matrix guided by entities, Attention-weighted guiding features For spatial dimension broadcasting operations, For masking key areas of the image, It is a step function. For batch normalization, This is the mask threshold for key regions of the image.

[0034] During the A5 step, and in the process of extracting the mask of key regions of the image based on the enhanced image features and aligned text enhancement features, the relevant neural network modules and hyperparameters are set as follows: The feature dimension d takes a value of 512; The number of kernels in the 3×3 convolutional layer is set to 256, the convolution stride is set to 1, and the padding method is Same padding. The momentum coefficient of the batch normalized layer is set to 0.99, and the epsilon value is set to 1e-5; The image key region mask threshold corresponding to the step function The value is 0.5, meaning that when the output of the Sigmoid function minus the calculated value of this threshold is greater than 0, the step function outputs 1, and otherwise outputs 0.

[0035] A6: Based on the image's key region mask, enhanced image features, domain-enhanced text features, and missing image semantic sets, output the security vulnerability analysis result text, including: A61: Based on the image's key region mask, enhanced image features, and domain-enhanced text features, extract the image focus features and compliance difference features of the key regions. The calculation method is as follows: ; ; ; in, Focusing on key image features, For global average pooling, The difference vector is the orthogonal projection. For vector norm, For compliance differences, It is the hyperbolic tangent function; A62: Based on compliance difference characteristics, key area image focus characteristics, and missing image semantic sets, output the security risk analysis result text. The calculation method is as follows: ; ; ; in, The confidence probability of a safety hazard. It is a fully connected layer. This is a preliminary safety hazard analysis report. For large language model decoders, For the safety hazard analysis results text, For large language models.

[0036] like Figure 2 As shown in this embodiment, a compliance inspection is conducted on an in-use distribution box at a construction site. First, images of the distribution box are acquired, and key visual features such as the box structure, incoming and outgoing line layout, and door connection are extracted. These features are then cross-modal aligned with the terminology dictionary and safety standard specifications in the distribution box field. Subsequently, by focusing on key areas and calculating compliance differences, missing defects and error defects are identified. Missing defects include the absence of fixed cable clips at the distribution box's incoming and outgoing line ports, while error defects include unsealed incoming and outgoing line ports, failure to connect the door and the box body with yellow-green wires, and direct contact between the incoming and outgoing lines and the box body. At the same time, each defect is associated with and matched with the corresponding safety standards, and finally, a safety hazard analysis result text containing the defect type, location, and basis for violation is output.

[0037] All neural network modules involved in the method described in this invention, including convolutional layers, bidirectional long short-term memory networks, Transformer encoders, multilayer perceptrons, attention mechanisms, classification prediction heads, regression prediction heads, and conditional random field decoders, are trained end-to-end. The model training process relies on preprocessed distribution box image data, distribution box safety standard specification text data, and corresponding manually labeled data. The specific training and optimization methods are as follows: To further improve the generalization performance of the model in the complex environment of waterway construction sites and to cope with diverse scenarios such as different water environments, different types of distribution boxes, different lighting conditions, and different working conditions, this invention optimizes the generalization ability of the dataset before model training. First, it constructs a dataset of distribution box images and text covering multiple scenarios, types, and working conditions, fully incorporating samples from different water environments, different distribution box structures, different lighting intensities, different shooting angles, and different construction conditions to compensate for insufficient data coverage in a single scenario. Simultaneously, it performs various data augmentation operations on the preprocessed distribution box image data. This includes lighting transformation, angle rotation, background replacement, and occlusion simulation, expanding the sample distribution space and enhancing the model's robustness to unknown scenarios. Furthermore, this embodiment introduces a domain-adaptive training mechanism, using measured data from distribution boxes collected in general scenarios as source domain data and actual data from distribution boxes specific to the waterway construction engineering field as target domain data for fusion training. In the model's shared feature layer, maximum mean difference (MMD) or domain adversarial methods are used to align feature distributions, reducing inter-domain differences and enabling the model to balance general detection capabilities with adaptability to waterway construction sites, thereby improving generalization performance and detection stability in complex scenarios. First, the preprocessed distribution box image data, safety standard specification text data, and corresponding entity annotation, hazard annotation, and location annotation label data are mixed and randomly divided into training set, validation set, and test set in a 7:2:1 ratio. The BERT pre-trained model is fine-tuned based on the safety specification text data of distribution boxes in waterway construction engineering for domain adaptation. The parameters of each embedding layer of the text are initialized based on the fine-tuning results. The weight parameters of the convolutional layer, bidirectional long short-term memory network, and Transformer encoder are randomly initialized using a standard normal distribution, and the bias parameters are initialized to 0. The parameters of modules such as multilayer perceptron, attention mechanism, classification prediction head, and regression prediction head are randomly initialized using a He normal distribution. To address the training requirements of different tasks in this invention, such as text semantic entity extraction, image semantic entity detection, cross-modal semantic alignment, missing class hazard identification, and key region mask generation, a weighted fusion of multiple loss functions is adopted to construct the overall loss function: the text-side semantic entity extraction task uses cross-entropy loss combined with the negative log-likelihood loss of the conditional random field decoder; the image-side semantic entity classification task uses cross-entropy loss, and the entity position regression task uses smoothing L1 loss; the cross-modal semantic alignment task uses contrastive loss; the missing class hazard identification task uses binary classification cross-entropy loss; and the key region mask generation task uses binary cross-entropy loss. The AdamW optimizer was selected for model training, with an initial learning rate of 1e-4 and a weight decay coefficient of 1e-5. The learning rate was scheduled using a cosine annealing decay strategy, which adaptively decayed the learning rate according to the number of training epochs. Dropout layers were set in modules such as the multilayer perceptron and bidirectional long short-term memory network, with a dropout rate of 0.1 to 0.3. An early stopping strategy was adopted throughout the model training process. If the model evaluation index on the validation set did not show significant improvement for 5 consecutive epochs, the model training was terminated immediately. The model training process uses mini-batch gradient descent for iterative training, with a batch size of 16. The gradients of the parameters of each neural network module are calculated using the backpropagation algorithm, and the parameters are iteratively updated based on the AdamW optimizer. There is no fixed value for the number of training rounds. The performance on the validation set is the core criterion. After each training round, the validation set data is input into the model and relevant evaluation metrics are calculated. If the metrics are better than the historical best values, the current model parameters are saved as the optimal model parameters.

[0038] It should be noted that the sequence numbers of the above embodiments of the present invention are merely for descriptive purposes and do not represent the superiority or inferiority of the embodiments. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, apparatus, article, or method. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0039] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0040] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A method for monitoring the electrical safety of distribution boxes in waterway construction projects based on machine vision, characterized in that, Includes the following steps: A1: Collect high-definition image data of the distribution box, text data of the electrical safety standards for the distribution box, and terminology data in the field of distribution boxes, and preprocess them separately to obtain preprocessed image data of the distribution box, preprocessed text data of the safety standards for the distribution box, and a dictionary of terminology in the field of distribution boxes. A2: Based on the preprocessed safety standard and specification text data of the distribution box, multi-scale fused text features are extracted, and then combined with the terminology dictionary of the distribution box domain, domain-enhanced text features are extracted, and then the set of text semantic entities is extracted. A3: Based on the preprocessed distribution box image data, extract global image features, calculate enhanced image features, entity category probability matrix, entity location coordinate matrix, and then extract the image semantic entity matrix; A4: Calculate the aligned text enhancement features and the missing image semantic set based on the text semantic entity set and the image semantic entity matrix; A5: Extract the mask of key regions in the image based on the enhanced image features and the aligned text enhancement features; A6: Based on the image key area mask, enhanced image features, domain-enhanced text features, and missing image semantic set, output the security risk analysis result text.

2. The method for monitoring the electrical safety of distribution boxes in waterway construction projects based on machine vision according to claim 1, characterized in that, Step A1 includes: A11: High-definition image data of power distribution boxes in waterway construction scenes are acquired by high-definition industrial cameras, and Gaussian filtering and scale normalization are performed to obtain pre-processed power distribution box image data. A12: Collect text data of electrical safety standards for distribution boxes in waterway construction scenarios. The data type is unstructured text data. Then, perform text cleaning and jieba word segmentation to obtain preprocessed text data of electrical safety standards for distribution boxes. A13: Collect terminology data in the field of distribution boxes, perform text cleaning processing, and store it in dictionary format to obtain a terminology dictionary for the field of distribution boxes.

3. The method for monitoring the electrical safety of distribution boxes in waterway construction projects based on machine vision according to claim 2, characterized in that, Step A2 includes: A21: Based on the preprocessed safety standard and specification text data for distribution boxes, character-level text encoding features, word-level text encoding features, and sentence-level text encoding features are extracted using convolutional layers, bidirectional long short-term memory networks, and Transformer encoders. The calculation method is as follows: ； ； ； ； in, These are the character-level embedding vectors, word-level embedding vectors, and sentence-level embedding vectors of the preprocessed distribution box safety standard specification text data. To pre-train a bidirectional large language model, This is the pre-processed text data of the safety standard specifications for the distribution box. Character-level text encoding features, It is a one-dimensional convolutional layer with residual connections. For word-level text encoding features, It is a bidirectional long short-term memory network. For sentence-level text encoding features, For Transformer encoders; A22: Based on character-level text encoding features, word-level text encoding features, and sentence-level text encoding features, multi-scale fused text features are extracted through a multi-scale semantic collaborative gating fusion mechanism. The calculation method is as follows: ； ； in, For multi-scale gated weight vectors, For the Softmax function, It is a multilayer perceptron. For feature splicing operations, To fuse text features at multiple scales, These are the first, second, and third feature components of the multi-scale gated weight vector, respectively. For Hadama accumulation, This is element-wise addition; A23: Based on multi-scale fused text features and a terminology dictionary for the distribution box domain, domain-enhanced text features are extracted through a terminology attention enhancement mechanism. The calculation method of the terminology attention enhancement mechanism is as follows: ； ； ； ； ； in, This is a coding vector for distribution box terminology. A dictionary of terms for the field of electrical distribution boxes. For the term attention weight matrix, For feature dimension, To match the feature matrix for the terms, For term gain weight, To enhance textual features for the domain, for function; A24: Based on domain-enhanced text features and multi-scale gated weight vectors, the text semantic entity matrix is ​​extracted using a conditional random field decoder. The calculation method is as follows: ； in, For text semantic entity matrix, For conditional random field decoders; A25: Using the Viterbi algorithm, the label sequence with the highest probability value is selected from the text semantic entity matrix to obtain the text semantic entity set. .

4. The method for monitoring the electrical safety of distribution boxes in waterway construction projects based on machine vision according to claim 3, characterized in that, Step A3 includes: A31: Input the preprocessed distribution box image data into the Transformer encoder, construct a feature sequence through convolutional mapping and positional encoding, and then use the Transformer encoder to extract global image features. The calculation method is as follows: ； in, For global image features, For flattening operation, It is a convolutional layer. The image data of the preprocessed distribution box. This is the position encoding matrix; A32: Based on global image features, feature enhancement is performed through a multi-scale feature fusion mechanism to obtain enhanced image features. The calculation method is as follows: ； ； in, Image mean features, For mean pooling operation, It is a vector of all 1s. These are the global image feature weight matrix and the image mean feature weight matrix, respectively. For enhanced image features, For layer normalization operation; A33: Based on the enhanced image features, the entity category probability matrix and entity location coordinate matrix are calculated using the classification prediction head and regression prediction head, respectively. Then, the image semantic entity matrix is ​​extracted. The calculation method is as follows: ； ； ； in, This is the entity category probability matrix. This is the category probability weight matrix. This is the class probability bias vector. This is the entity's position coordinate matrix. This is the position coordinate weight matrix. The position coordinate offset vector, For image semantic entity matrix, For indicator functions, To obtain the maximum value, The confidence threshold for semantic entities in an image.

5. The method for monitoring the electrical safety of distribution boxes in waterway construction projects based on machine vision, as described in claim 4, is characterized in that... The A4 step includes: A41: Based on the text semantic entity set and the image semantic entity matrix, cross-modal semantic alignment is performed through an attention mechanism to obtain aligned text enhancement features; A42: Output the missing image semantic set based on the aligned text enhancement features.

6. The method for monitoring the electrical safety of distribution boxes in waterway construction projects based on machine vision, as described in claim 5, is characterized in that... The calculation process for the missing image semantic set in step A4 includes: First, the text semantic entity set is processed by BERT. The processing result, along with the image semantic entity matrix, is input into an attention mechanism for cross-modal semantic alignment to obtain aligned text augmentation features. Then, the BERT processing result of the text semantic entity set is concatenated with the aligned text augmentation features, and the concatenated result is processed by a multilayer perceptron and input into a sigmoid function to obtain an entity existence score vector. Finally, the existence score of each text entity in the entity existence score vector is compared with a text entity existence score threshold, and text semantic entities with existence scores below the threshold in the text semantic entity set are selected to obtain the missing image semantic set.

7. The method for monitoring the electrical safety of distribution boxes in waterway construction projects based on machine vision according to claim 6, characterized in that, The process of extracting the mask for key regions of the image in step A5 includes: First, the aligned text enhancement features and the enhanced image features are multiplied by an inner product. The result is divided by the square root of the feature dimension and then processed by the Softmax function to obtain the entity-guided spatial attention matrix. Next, a spatial dimension broadcasting operation is performed on the entity-guided spatial attention matrix. The broadcasting result is then multiplied by the enhanced image features using a Hadamard product, and processed by a 3×3 convolutional layer to obtain attention-weighted guided features. Finally, the attention-weighted guided features are batch-normalized, and the result is input into the Sigmoid function. The difference between the output of the Sigmoid function and the image key region mask threshold is calculated, and the difference is input into a step function to obtain the image key region mask.

8. The method for monitoring the electrical safety of distribution boxes in waterway construction projects based on machine vision, as described in claim 7, is characterized in that... Step A6 includes: A61: Extract the image focus features and compliance difference features of key regions based on the image key region mask, enhanced image features, and domain-enhanced text features; A62: Output the text of the security risk analysis results based on the compliance difference characteristics, the image focus characteristics of key areas, and the missing image semantic set.

9. The method for monitoring the electrical safety of distribution boxes in waterway construction projects based on machine vision, as described in claim 8, is characterized in that... Step A6 includes: First, perform a Hadamard product between the enhanced image features and the image key region mask, and then perform global average pooling on the product result to obtain the image focus features of the key region. Next, calculate the inner product of the key region image focus feature and the aligned text enhancement feature. Divide the inner product result by the square of the vector norm of the aligned text enhancement feature. Then multiply the calculated result with the aligned text enhancement feature. Finally, subtract the multiplication result from the key region image focus feature to obtain the orthogonal projection difference vector. Next, the difference between the orthogonal projection difference vector and the focus feature of the key region image minus the aligned text enhancement feature is concatenated. The concatenated result is then processed by a multilayer perceptron and input into the hyperbolic tangent function to obtain the compliance difference feature. The compliance difference features are then processed through a fully connected layer to obtain the security risk confidence probability. The security risk confidence probability is then concatenated with the compliance difference features and input into the large language model decoder to obtain the preliminary security risk analysis result text. Finally, the missing image semantic set and the preliminary security risk analysis result text are input into the large language model for processing to obtain the security risk analysis result text.

Images