Valve hall equipment identification method based on multi-scale feature fusion and semantic matching

Abstract

The invention discloses a valve hall equipment identification method based on multi-scale feature fusion and semantic matching, and the method comprises the steps: collecting a valve hall image, and carrying out the preprocessing of the image; and inputting the preprocessed image into a multi-scale feature extraction network based on a feature pyramid network to generate a multi-scale visual feature map. Constructing an equipment identification model, training a visual projection module and a semantic projection module in an end-to-end manner through an alignment training mechanism based on adaptive dynamic contrast learning, generating equipment category information through a region identification module based on the trained visual projection module, and combining a visual feature vector and the category information as input of a positioning module; and performing segmentation task training. And constructing structured text semantic description for each device of the valve hall, generating semantic feature vectors through a semantic projection module, constructing a semantic knowledge base, and realizing visual-semantic matching recognition. The method can effectively solve the problems of large scale difference of valve hall equipment, complex environment, insufficient semantic understanding and the like, and improves the recognition precision and robustness.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image recognition, and particularly relates to a valve hall equipment recognition method based on multi-scale feature fusion and semantic matching. BACKGROUND

[0002] The valve hall of a converter station is a core link of a DC power transmission system, and the running state of internal equipment (such as a valve tower, a valve-side bushing, a lightning arrester, a lead joint, a crimping bolt, etc.) directly relates to the safety and stability of the entire power grid. Real-time and accurate inspection and recognition of the valve hall equipment are of great significance for guaranteeing the safety of power grid operation.

[0003] Traditional manual inspection is highly dependent on the experience of inspection personnel and has problems such as missed inspection and misjudgment. Meanwhile, the high-voltage, closed and space-limited environment in the valve hall makes the inspection have great safety risks. In the prior art, CN115830302A discloses a multi-scale feature extraction and fusion power distribution network equipment positioning and recognition method, and the specific steps are as follows: obtaining power distribution network equipment images under different scenes and performing preprocessing, and dividing a training set, a verification set and a test set; constructing a target detection network composed of a multi-scale Transformer feature extraction backbone network, a pyramid multi-scale feature fusion module and a classification detection prediction layer; after learning the weight parameters of the training set and further optimizing and adjusting the verification set, the final detection result of the power distribution network equipment is obtained by the test set.

[0004] However, the method has the following limitations: the method mainly stays at pixel level or appearance feature recognition, and lacks deep semantic understanding of equipment functions, structural features and spatial relationships between components. In the case of equipment appearance abnormalities or partial occlusion, the recognition reliability sharply decreases, and accurate recognition of complex scenes cannot be achieved.

[0005] Therefore, there is an urgent need for a method that can simultaneously adapt to multi-scale changes of equipment, has semantic understanding ability, and realizes high-precision and robust recognition in complex environments, so as to meet the safety, reliability and intelligentization requirements of valve hall equipment inspection and operation and maintenance. SUMMARY

[0006] The purpose of the present application is to overcome the defects of the prior art and provide a valve hall equipment recognition method based on multi-scale feature fusion and semantic matching.

[0007] The purpose of the present application can be achieved by the following technical solutions: The present application provides a valve hall equipment recognition method based on multi-scale feature fusion and semantic matching, comprising the following steps: obtaining a valve hall image to be recognized and performing preprocessing; The preprocessed valve hall image is input into a multi-scale feature extraction network based on a feature pyramid network, and multi-scale visual feature maps are generated by fusing shallow, medium and deep features from top to bottom. A device recognition model is constructed and trained to obtain a trained device recognition model; the device recognition model includes a visual projection module, a semantic projection module, a region recognition module, and a localization module. Structured text semantic descriptions are constructed for each device in the valve hall, and the structured text semantic descriptions of each device are converted into corresponding semantic feature vectors based on the semantic projection module of the trained device recognition model, thus constructing a semantic knowledge base for the devices in the valve hall. The multi-scale visual feature map is input into the trained device recognition model for processing. Based on the semantic knowledge base, the valve hall device category, category similarity score, and corresponding bounding box are output.

[0008] Furthermore, the preprocessing includes: converting the acquired valve hall image to grayscale; applying Gaussian filtering to the grayscale image; and performing histogram equalization on the filtered image to obtain the preprocessed valve hall image.

[0009] Furthermore, the multi-scale feature extraction network based on the feature pyramid network includes a backbone convolutional neural network, a feature pyramid fusion module, and a CBAM attention module; The processing steps of the multi-scale feature extraction network include: The preprocessed valve hall image is input into the backbone convolutional neural network to extract feature maps of different depths. ,in, i Indicates the network stage index. These correspond to shallow, intermediate, and deep layer features, respectively. Feature maps of each stage The input feature pyramid fusion module achieves multi-scale feature fusion through top-down paths and lateral connections, generating a fused feature map. , is represented as: in, This indicates the backbone convolutional neural network's first... i Feature maps output by each stage; The feature pyramid fusion module represents the fused first feature. i Multi-scale feature maps; express Convolution operation; This indicates a bilinear interpolation upsampling operation; the final obtained... As a fusion feature map ; The fused feature map The input to the CBAM attention module is weighted by channel and spatial attention to obtain multi-scale visual feature maps. , means as follows: in, This represents the channel weights generated by the channel attention unit; This represents the global max pooling operation; This indicates a global average pooling operation; Indicates multilayer perceptron operation; This represents the Sigmoid activation function; This indicates element-wise multiplication. express Convolution operation; This represents the spatial weights generated by the spatial attention unit; Represents multi-scale visual feature maps.

[0010] Furthermore, the visual projection module is a module composed of a convolutional neural network and a fully connected layer, used to convert multi-scale visual feature maps into fixed-dimensional visual feature vectors. The semantic projection module is a Sentence-BERT encoding model used to convert the structured textual semantic description of the valve hall equipment into a corresponding semantic feature vector. The region recognition module is used to calculate the similarity between the visual feature vector generated by the visual projection module and each semantic feature vector in the semantic knowledge base, select the device category with the highest similarity as the recognition result, and use the similarity as the output category similarity score. The localization module is a segmentation head based on Faster R-CNN. Based on the visual feature vector generated by the visual projection module and the device category generated by the region recognition module, it generates an accurate device location bounding box.

[0011] Furthermore, the training process for the device recognition model includes: Based on the alignment training mechanism of contrastive learning, the visual projection module and semantic projection module of the device recognition model are aligned and trained to obtain the trained visual projection module and semantic projection module. Based on the trained visual projection module, the device category information generated by the region recognition module is used to train the localization module of the device recognition model for segmentation tasks, thereby obtaining the trained localization module.

[0012] Furthermore, the alignment training mechanism based on contrastive learning performs alignment training on the visual projection module and semantic projection module of the device recognition model to obtain the trained visual projection module and semantic projection module, specifically including: Images of various devices in the valve hall are acquired, and multi-scale visual feature maps of these devices are obtained through a multi-scale feature extraction network. The devices include valve towers, valve side bushings, surge arresters, lead wire connectors, and crimping bolts. A structured text semantic description is constructed for each device, which includes the device name, device function, device structural features, and spatial relationship between devices; The first training dataset is constructed by using multi-scale visual feature maps of images from various devices and their corresponding structured text semantic descriptions as samples. ;in, Indicates the first i Multi-scale visual feature maps of individual device images; Indicates the first i Structured textual semantic description of each device image; Indicates the total number of samples; The multi-scale visual feature maps and structured text semantic descriptions from the first training dataset are input into the visual projection module and the semantic projection module, respectively, to generate corresponding visual feature vectors and semantic feature vectors. Based on the generated visual feature vectors and semantic feature vectors, the visual projection module and semantic projection module are trained end-to-end according to the first loss function to obtain the trained visual projection module and semantic projection module.

[0013] Furthermore, the first loss function is an adaptive dynamic contrastive learning loss function, the formula of which is: in, The first loss function; Indicates the first i Visual feature vectors generated from device images via a visual projection module; Indicates the first i The semantic feature vector generated by the semantic projection module corresponding to each device; , Preset weighting coefficients; Cosine similarity; Indicates the training of the first t Samples from each batch i and j Adaptive dynamic inter-class negative sample weights; This indicates the current training batch number.

[0014] Furthermore, the adaptive dynamic negative sample weights between categories are calculated using the following formula: in, For adaptive weight adjustment coefficients, This is the historical average, representing the value up to the [number]th [number]. One batch, samples i Visual feature vectors and samples j The historical average of the cosine similarity of the semantic feature vectors.

[0015] Furthermore, based on the trained visual projection module, the device category information generated by the region recognition module is used to train the localization module of the device recognition model for a segmentation task, thereby obtaining the trained localization module, specifically including: Valve hall images were acquired using multi-scale methods (far, medium, and near), and the acquired images were filtered, including removing blurry, severely occluded, or abnormally lit images. For the selected valve hall images, the true bounding boxes of each valve hall image are obtained through manual annotation. ;in, Indicates the coordinates of the top-left corner of the bounding box; Indicates the width and height of the bounding box; For the selected valve hall images, a multi-scale visual feature map of the valve hall image is obtained through a multi-scale feature extraction network. The multi-scale visual feature map is then input into the trained visual projection module to obtain the corresponding visual feature vector. ; The visual feature vector is input into the region recognition module, which calculates the cosine similarity between the visual feature vector and each semantic feature vector in the semantic knowledge base. The device category with the highest cosine similarity is used as the device category information for each valve hall image. ; The visual feature vectors of each valve chamber image are combined with equipment category information to form the input features for the positioning module. ; Input features A second training dataset was constructed using the ground truth bounding boxes as samples. ;in, This represents the total number of samples in the second training dataset; The second training dataset Input features The localization module is input for segmentation task training and outputs predicted bounding boxes. Based on the predicted bounding boxes and the ground truth bounding boxes, the localization module is trained using a second loss function to obtain the trained localization module.

[0016] Furthermore, the second loss function is formulated as follows: in, For the second loss function, The first number predicted by the positioning module is... i Predicted bounding boxes for each valve chamber image; Indicates the first i The intersection-union ratio (IoU) of the predicted bounding boxes and the ground truth bounding boxes for each sample.

[0017] The acquisition of valve hall images using a multi-scale approach (far, mid, and near) specifically includes: Mobile acquisition equipment is used to acquire images of the valve hall according to a preset inspection path. During the acquisition process, panoramic images of the valve tower are acquired by shooting from a distance, mesoscopic images of individual equipment are acquired by shooting from a medium distance, and close-up images of small components are acquired by shooting from a close distance. The individual equipment includes surge arresters and valve side bushings. The small components include lead wire connectors and crimping bolts. The acquisition is repeated at different time periods and under different lighting conditions.

[0018] Compared with the prior art, the present invention has the following advantages: (1) In the prior art, valve hall equipment inspection is highly dependent on manual operation, which is inefficient, prone to missed detections and misjudgments. In addition, the valve hall environment is high-pressure and closed, posing safety hazards. The present invention uses a mobile acquisition device to acquire images of the valve hall according to a preset inspection path, thereby replacing manual inspection, improving inspection efficiency, reducing the risk of missed detections and misjudgments, and reducing the safety hazards of inspection in a high-pressure closed environment. It realizes safe and efficient valve hall equipment identification and inspection.

[0019] (2) In the prior art, the size of equipment in the valve hall varies greatly. From the valve tower to the lead wire connector, traditional image processing or early deep learning methods are difficult to take into account both the overall structure of large targets and the fine features of small targets, resulting in insufficient recognition accuracy. This invention integrates shallow, medium and deep features from top to bottom through a multi-scale feature extraction network, generates multi-scale visual feature maps by combining a feature pyramid network, and performs channel and spatial attention weighting through the CBAM module, so that the model can capture the global structural features of large targets and the detailed features of small targets at the same time, thereby significantly improving the detection and recognition accuracy of equipment of different scales and realizing accurate recognition of equipment of various scales in the valve hall.

[0020] (3) In the prior art, the valve hall environment is complex, with dense equipment, occlusion, changes in lighting, and differences in viewing angle, resulting in poor robustness of existing recognition models and easy misidentification or omission. This invention constructs a semantic knowledge base for equipment and uses the visual feature vectors generated by the visual projection module and the semantic feature vectors generated by the semantic projection module to perform comparative learning and alignment training, thereby realizing the shared latent space mapping of visual and semantic cross-modal features; during the recognition process, the region recognition module calculates based on cosine similarity to realize semantic matching and contextual reasoning for low-confidence regions, and the localization module generates accurate bounding boxes, thereby enhancing the robustness of the model in complex environments and effectively solving the recognition instability problems caused by occlusion, uneven lighting, and changes in viewing angle.

[0021] (4) In the prior art, existing methods mostly focus on pixel-level or appearance-level recognition, lacking semantic understanding of equipment functions, structural features, and spatial relationships between components. Recognition reliability drops sharply when the equipment has an abnormal appearance or is partially occluded. This invention constructs a structured textual semantic description for valve hall equipment, including equipment name, function, structural features, and spatial relationships between components. It then maps the textual semantics to a shared latent space using a semantic projection module and performs comparative learning using visual feature vectors, achieving visual-semantic alignment and enabling the model to possess deep semantic understanding capabilities. This technical feature allows the model to accurately identify equipment categories even when the equipment is partially occluded or has an abnormal appearance, significantly improving recognition reliability and interpretability.

[0022] (5) In the prior art, existing methods are insufficient in distinguishing the similarity of different types of devices, and category confusion easily occurs during training, leading to a decrease in recognition accuracy. This invention introduces adaptive dynamic negative sample weights between categories, updates the distinction weights between categories in real time during training, and assigns higher distinction weights to easily confused categories. Combined with adaptive dynamic contrastive learning loss function for training, this maximizes the similarity of visual and semantic feature vectors of similar devices and minimizes the similarity of dissimilar devices, thereby effectively improving the model's ability to distinguish similar devices and achieving high-precision valve hall device recognition.

[0023] (6) Although simple visual feature vectors contain spatial and texture information of images, they lack category semantic constraints, which can easily lead to inaccurate bounding box predictions in dense or complex backgrounds. Therefore, this invention combines the visual feature vectors generated by the visual projection module with the device category information generated by the region recognition module as input to the localization module. By combining the device category information with the visual feature vectors, the localization module not only considers the visual features of the image when generating bounding boxes, but also uses category semantic constraints to limit the target range, thereby improving localization accuracy and reliability. This enables the localization module to generate more accurate and stable device bounding boxes in complex environments (such as dense equipment, partial occlusion, or changes in lighting), avoiding localization deviations caused by category confusion, and providing high-quality structured spatial information for subsequent intelligent operation and maintenance, fault diagnosis, and other applications. Attached Figure Description

[0024] Figure 1 This is a flowchart of the valve hall equipment identification method according to an embodiment of the present invention; Figure 2 This is a model diagram of the valve hall equipment identification system according to an embodiment of the present invention. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0026] Example 1: This embodiment provides a valve hall equipment identification method based on multi-scale feature fusion and semantic matching, such as... Figure 1 As shown, it includes the following steps: Step S1: Obtain the image of the valve chamber to be identified and perform preprocessing; Preprocessing includes: grayscale conversion: converting color images to grayscale images to reduce computational complexity and minimize the impact of illumination variations; Gaussian filtering: applying Gaussian filtering to grayscale images to eliminate random noise, improve image quality, and facilitate subsequent feature extraction; and histogram equalization: performing histogram equalization on the filtered images to enhance image contrast, making key features of the device more prominent, thereby improving the effectiveness of multi-scale feature extraction.

[0027] Step S2: Input the preprocessed valve hall image into a multi-scale feature extraction network based on the feature pyramid network, and generate a multi-scale visual feature map by fusing shallow, medium and deep features from top to bottom; Among them, the multi-scale feature extraction network based on the feature pyramid network includes a backbone convolutional neural network, a feature pyramid fusion module, and a CBAM attention module; The processing steps of a multi-scale feature extraction network include: The preprocessed valve hall image is input into the backbone convolutional neural network to extract feature maps of different depths. ,in, i Indicates the network stage index. These correspond to shallow, intermediate, and deep layer features, respectively. Feature maps of each stage The input feature pyramid fusion module achieves multi-scale feature fusion through top-down paths and lateral connections, generating a fused feature map. , is represented as: in, This indicates the backbone convolutional neural network's first... i Feature maps output by each stage; The feature pyramid fusion module represents the fused first feature. i Multi-scale feature maps; express Convolution operation; This indicates a bilinear interpolation upsampling operation; the final obtained... As a fusion feature map ; The fused feature map The input to the CBAM attention module is weighted by channel and spatial attention to obtain multi-scale visual feature maps. , means as follows: in, This represents the channel weights generated by the channel attention unit; This represents the global max pooling operation; This indicates a global average pooling operation; Indicates multilayer perceptron operation; This represents the Sigmoid activation function; This indicates element-wise multiplication. express Convolution operation; This represents the spatial weights generated by the spatial attention unit; Represents multi-scale visual feature maps.

[0028] In step S2, the preprocessed valve hall image is first processed by a backbone convolutional neural network to extract feature maps of different depths. Shallow features retain rich spatial details, such as the edges and textures of small components; mid-level features consider both local structure and semantic information; and deep features primarily contain high-level semantic information, such as the overall shape and functional features of the valve tower or surge arrester. This multi-layered feature extraction ensures that devices of different scales in the image are fully represented, which is impossible to achieve by directly inputting the image into a visual projection module, as features at a single scale often struggle to capture both the fine details of small components and the overall structure of large equipment.

[0029] The feature pyramid fusion module fuses shallow, medium, and deep features through a top-down path and lateral connections to generate a fused feature map, thereby simultaneously achieving high spatial resolution and strong semantic expressive power. This step enhances the model's ability to identify devices with vastly different scales in a valve hall (such as those ranging from massive valve towers to tiny lead connectors), solving the problem in existing technologies where large and small targets are difficult to simultaneously address.

[0030] The CBAM attention module applies channel and spatial attention weights to the fused feature map. Channel attention highlights key channel features, while spatial attention focuses on the area where the device is located, generating multi-scale visual feature maps. This further enhances the effective information density of the feature maps, making the features received by the visual projection module more prominent in terms of the key features of the device itself, while suppressing background interference or redundant information, thus improving the accuracy and robustness of valve hall device recognition.

[0031] The process of first processing the image through a multi-scale feature extraction network instead of directly inputting it into the visual projection module is to achieve adaptive feature extraction of devices at different scales in the valve hall image, taking into account both details and global semantics, while enhancing the effectiveness of feature expression. This provides a high-quality visual feature foundation for subsequent visual-semantic alignment and localization modules, thereby significantly improving the accuracy and robustness of device recognition.

[0032] Step S3: Construct and train the device recognition model to obtain the trained device recognition model; the device recognition model includes a visual projection module, a semantic projection module, a region recognition module, and a localization module; Among them, the visual projection module is a module composed of a convolutional neural network and a fully connected layer, which is used to convert multi-scale visual feature maps into fixed-dimensional visual feature vectors; The semantic projection module is a Sentence-BERT encoding model used to convert the structured textual semantic description of the valve hall equipment into corresponding semantic feature vectors; The region recognition module is used to calculate the similarity between the visual feature vector generated by the visual projection module and each semantic feature vector in the semantic knowledge base, select the device category with the highest similarity as the recognition result, and output the similarity as the category similarity score. The localization module uses a segmentation head based on Faster R-CNN. Based on the visual feature vector generated by the visual projection module and the device category generated by the region recognition module, it generates an accurate device location bounding box.

[0033] The device recognition model training process includes: Based on the alignment training mechanism of contrastive learning, the visual projection module and semantic projection module of the device recognition model are aligned and trained to obtain the trained visual projection module and semantic projection module, specifically including: Images of various devices in the valve hall are acquired, and multi-scale visual feature maps of these devices are obtained through a multi-scale feature extraction network. The devices include valve towers, valve side bushings, surge arresters, lead wire connectors, and crimping bolts. A structured text semantic description is constructed for each device. The structured text semantic description includes the device name, device function, device structural features, and spatial relationships between devices. The first training dataset is constructed by using multi-scale visual feature maps of images from various devices and their corresponding structured text semantic descriptions as samples. ;in, Indicates the first i Multi-scale visual feature maps of individual device images; Indicates the first i Structured textual semantic description of each device image; Indicates the total number of samples; The multi-scale visual feature maps and structured text semantic descriptions from the first training dataset are input into the visual projection module and the semantic projection module, respectively, to generate corresponding visual feature vectors and semantic feature vectors. Based on the generated visual feature vectors and semantic feature vectors, the visual projection module and semantic projection module are trained end-to-end according to the first loss function to obtain the trained visual projection module and semantic projection module.

[0034] The first loss function is the adaptive dynamic contrastive learning loss function, and the formula is: in, The first loss function; Indicates the first i Visual feature vectors generated from device images via a visual projection module; Indicates the first iThe semantic feature vector generated by the semantic projection module corresponding to each device; , Preset weighting coefficients; Cosine similarity; Indicates the training of the first t Samples from each batch i and j Adaptive dynamic inter-class negative sample weights; This indicates the current training batch number.

[0035] Adaptive dynamic inter-class negative sample weights, the formula is: in, For adaptive weight adjustment coefficients, This is the historical average, representing the value up to the [number]th [number]. One batch, sample i Visual feature vectors and samples j The historical average of the cosine similarity of the semantic feature vectors.

[0036] In the first training process of this invention, end-to-end training of visual feature vectors and semantic feature vectors achieves the alignment mapping between the visual projection module and the semantic projection module. Specifically, this training uses an adaptive dynamic contrastive learning loss function to maximize the cosine similarity between visual feature vectors and semantic feature vectors of the same device category, while minimizing the cosine similarity between feature vectors of different categories, thereby accurately mapping visual information to semantic information in the shared latent space. The adaptive dynamic negative sample weights between categories can be dynamically adjusted according to the degree of confusion between different categories during training, updating the historical average similarity in real time. This appropriately amplifies the impact of more easily confused categories on model training, improving the model's ability to distinguish similar devices. Compared to fixed-weight contrastive learning methods, this mechanism is more adaptable to the complex visual and semantic differences between device categories, improving the accuracy and robustness of feature alignment. The technical effect of this training process is to ensure that visual features and semantic features are highly consistent in the shared latent space, enabling the subsequent region recognition module to perform accurate category matching based on visual features and the semantic knowledge base, thus providing reliable category prior information for the localization module. For the overall technical solution, this step is the core foundation for achieving deep visual-semantic fusion and improving the recognition accuracy and robustness of multi-scale and multi-category valve hall equipment.

[0037] Based on the trained visual projection module, and using the device category information generated by the region recognition module, the localization module of the device recognition model is trained using a segmentation task to obtain the trained localization module, which specifically includes: Valve hall images were acquired using multi-scale methods (far, medium, and near), and the acquired images were filtered, including removing blurry, severely occluded, or abnormally lit images. For the selected valve hall images, the true bounding boxes of each valve hall image are obtained through manual annotation. ;in, Indicates the coordinates of the top-left corner of the bounding box; Indicates the width and height of the bounding box; For the selected valve hall images, a multi-scale visual feature map of the valve hall image is obtained through a multi-scale feature extraction network. The multi-scale visual feature map is then input into the trained visual projection module to obtain the corresponding visual feature vector. ; The visual feature vector is input into the region recognition module, which calculates the cosine similarity between the visual feature vector and each semantic feature vector in the semantic knowledge base. The device category with the highest cosine similarity is used as the device category information for each valve hall image. ; The visual feature vectors of each valve chamber image are combined with equipment category information to form the input features for the positioning module. ; Input features A second training dataset was constructed using the ground truth bounding boxes as samples. ;in, This represents the total number of samples in the second training dataset; The second training dataset Input features The localization module is input for segmentation task training and outputs predicted bounding boxes. Based on the predicted bounding boxes and the ground truth bounding boxes, the localization module is trained using a second loss function to obtain the trained localization module.

[0038] The second loss function is expressed as follows: in, For the second loss function, The first number predicted by the positioning module is... i Predicted bounding boxes for each valve chamber image; Indicates the first i The intersection-union ratio (IoU) of the predicted bounding boxes and the ground truth bounding boxes for each sample.

[0039] In the training process of the positioning module of this invention, the device category information generated by the trained visual projection module and region recognition module is combined with the visual feature vector as the input feature of the positioning module. The main purpose of this design is to provide the positioning module with category prior information, so that it can rely not only on the visual features themselves in the segmentation task, but also combine the semantic category of the device, which helps to distinguish devices with similar appearances but different categories, thereby improving positioning accuracy. Combined with category information, the positioning module can more accurately distinguish devices with large differences in size and appearance, such as valve towers, surge arresters, and lead connectors. In the case of uneven lighting, occlusion, or complex background, category information provides additional discrimination clues, reducing false detections and false negatives. By fusing visual features and category information into the positioning module, the identification and positioning of valve hall devices form a closed loop, improving the overall performance of the entire multi-scale feature fusion and semantic matching technology solution.

[0040] Step S4: Construct structured text semantic descriptions for each device in the valve hall, and convert the structured text semantic descriptions of each device into corresponding semantic feature vectors based on the semantic projection module of the trained device recognition model, thereby constructing a semantic knowledge base for the valve hall devices. Specifically, this includes: The system first identifies the categories of equipment in the valve hall, such as valve towers, valve side bushings, surge arresters, lead wire connectors, and crimp bolts. Then, it constructs a structured text semantic description for each category, including the equipment name, function, main structural features, and spatial relationships with other equipment. This ensures accurate semantic representation of the equipment's characteristics and interrelationships. The structured text semantic description is then input into a trained semantic projection module (such as the Sentence-BERT encoding model) to vectorize the text information, generating fixed-dimensional semantic feature vectors. Finally, the semantic feature vectors of all equipment categories are stored and organized into a semantic knowledge base for the valve hall equipment. This knowledge base serves as a reference for calculating the similarity between visual and semantic features during subsequent equipment identification, achieving accurate semantic matching of valve hall equipment categories.

[0041] Step S5: Input the multi-scale visual feature map into the trained device recognition model for processing. Based on the semantic knowledge base, output the valve hall device category, category similarity score, and its corresponding bounding box, specifically including: The multi-scale visual feature map generated in step S2 is input into the trained visual projection module, and processed by a convolutional neural network and a fully connected layer to generate a fixed-dimensional visual feature vector. The generated visual feature vector is input into the region recognition module. The region recognition module calculates the cosine similarity between the visual feature vector and the semantic feature vector of each device category in the semantic knowledge base, selects the device category with the highest similarity as the recognition result, and uses the similarity value as the confidence score of that category. The visual feature vector is combined with the identified equipment category information and then input into the localization module. The localization module, based on the Faster R-CNN segmentation head, combines the visual features and category information to generate the precise bounding box of the valve hall equipment. The output includes the identification category of each valve chamber device, the corresponding category similarity score, and the precise device location bounding box, enabling accurate and robust identification and positioning of multi-scale devices, and providing reliable data support for subsequent equipment status analysis or automatic inspection.

[0042] Example 2: This embodiment provides a valve hall equipment recognition system based on multi-scale feature fusion and semantic matching, such as... Figure 2 As shown, it includes: The acquisition unit is used to acquire valve hall images at multiple scales (far, medium, and near) and preprocess the acquired images, including grayscale conversion, Gaussian filtering, and histogram equalization, to obtain valve hall images suitable for subsequent feature extraction. The multi-scale feature extraction unit, built on a feature pyramid network, includes a backbone convolutional neural network, a feature pyramid fusion module, and a CBAM attention module. It is used to extract features from the preprocessed valve hall image, generate a multi-scale visual feature map that fuses shallow, medium, and deep features, and highlight key feature regions through channel and spatial attention weighting. The device recognition model includes a visual projection module, a semantic projection module, a region recognition module, and a localization module. The visual projection module is used to convert multi-scale visual feature maps into fixed-dimensional visual feature vectors. The semantic projection module (such as the Sentence-BERT encoding model) is used to map the structured textual semantic description of the valve hall device into semantic feature vectors. The region recognition module is used to calculate the similarity between the visual feature vector and the semantic feature vectors of each category in the semantic knowledge base and output the recognition category and similarity score. The localization module generates accurate device location bounding boxes based on the visual feature vector and category information using the Faster R-CNN segmentation head. A semantic knowledge base is used to store the semantic feature vectors of each device in the valve hall, so that the area recognition module can perform visual-semantic matching during the recognition process. The first training unit is used to perform alignment training on the visual projection module and semantic projection module of the device recognition model based on adaptive dynamic contrastive learning. By minimizing the adaptive dynamic contrastive learning loss function, the cosine similarity between the visual feature vector and the semantic feature vector of the same type of device is maximized, while the cosine similarity between the different types of devices is minimized. This achieves the alignment of visual and semantic features in the shared latent space, thereby improving the accuracy and robustness of valve hall device recognition. The second training unit is used to train the localization module for segmentation tasks based on the device category information generated by the region recognition module after training the visual projection module. By minimizing the intersection-union ratio loss function between the predicted bounding box and the real bounding box, the trained localization module is obtained, thereby achieving accurate localization and segmentation of the valve hall equipment and improving the spatial accuracy of the recognition results.

[0043] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0044] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A valve hall equipment identification method based on multi-scale feature fusion and semantic matching, characterized in that, The method comprises the following steps: obtaining a valve hall image to be recognized and performing preprocessing; inputting the preprocessed valve hall image into a multi-scale feature extraction network based on a feature pyramid network, and generating a multi-scale visual feature map through the top-down fusion of shallow, middle and deep features; constructing a device recognition model and training the same to obtain a trained device recognition model; the device recognition model comprises a visual projection module, a semantic projection module, a region recognition module and a positioning module; constructing a structured text semantic description for each device in the valve hall, and converting the structured text semantic description of each device into a corresponding semantic feature vector based on the semantic projection module of the trained device recognition model to construct a semantic knowledge base of the valve hall device; inputting the multi-scale visual feature map into the trained device recognition model for processing, and outputting the valve hall device category, category similarity score and corresponding bounding box according to the semantic knowledge base.

2. The valve hall equipment identification method based on multi-scale feature fusion and semantic matching according to claim 1, characterized in that, The preprocessing comprises: performing grayscale processing on the collected valve hall image; applying Gaussian filtering to the grayscale image; and performing histogram equalization on the filtered image to obtain the preprocessed valve hall image.

3. The valve hall equipment identification method based on multi-scale feature fusion and semantic matching according to claim 1, characterized in that, The multi-scale feature extraction network based on the feature pyramid network comprises a backbone convolutional neural network, a feature pyramid fusion module and a CBAM attention module. The processing process of the multi-scale feature extraction network comprises: The pretreated valve hall image is input into a backbone convolutional neural network to extract feature maps at different depths wherein, i denotes a network stage index, respectively correspond to shallow, middle and deep features; The feature maps of each stage are fused by the input feature pyramid fusion module The input feature pyramid fusion module realizes multi-scale feature fusion through a top-down path and a horizontal connection to generate a fused feature map , which is represented as: wherein, represents the feature map output by the main convolutional neural network in the i stage; represents the multi-scale feature map output by the feature pyramid fusion module in the i stage; represents a convolution operation; represents a bilinear interpolation up-sampling operation; and the finally obtained is taken as the fusion feature map ; The fused feature map The input to the CBAM attention module is weighted by channel and spatial attention to obtain multi-scale visual feature maps. , means as follows: wherein, represents a channel weight generated by a channel attention unit; represents a global max pooling operation; represents a global average pooling operation; represents a multi-layer perceptron operation; represents a Sigmoid activation function; represents an element-wise multiplication operation; represents a convolution operation; represents a spatial weight generated by a spatial attention unit; represents a multi-scale visual feature map.

4. The valve hall equipment identification method based on multi-scale feature fusion and semantic matching according to claim 1, characterized in that, The visual projection module is a module composed of a convolutional neural network and a fully connected layer, which is used to convert the multi-scale visual feature map into a fixed-dimensional visual feature vector; The semantic projection module is a Sentence-BERT encoding model, which is used to convert the structured text semantic description of the valve hall device into a corresponding semantic feature vector; The region recognition module is used to calculate the similarity between the visual feature vector generated by the visual projection module and each semantic feature vector in the semantic knowledge base, select the device category with the highest similarity as the recognition result, and output the similarity as the category similarity score; The positioning module is a segmentation head based on Faster R-CNN, which generates an accurate device position bounding box based on the visual feature vector generated by the visual projection module and the device category generated by the region recognition module.

5. The valve hall equipment identification method based on multi-scale feature fusion and semantic matching according to claim 1, characterized in that, The training process of the device recognition model comprises: aligning and training the visual projection module and the semantic projection module of the device recognition model based on a contrast learning alignment training mechanism to obtain the trained visual projection module and the semantic projection module; based on the trained visual projection module, performing segmentation task training on the positioning module of the device recognition model through the device category information generated by the region recognition module to obtain the trained positioning module.

6. The valve hall equipment identification method based on multi-scale feature fusion and semantic matching according to claim 5, characterized in that, The contrast learning alignment training mechanism aligns and trains the visual projection module and the semantic projection module of the device recognition model to obtain the trained visual projection module and the semantic projection module, and specifically comprises: obtaining images of each device in the valve hall, and obtaining multi-scale visual feature maps of the images of each device in the valve hall through a multi-scale feature extraction network; the devices include valve towers, valve side sleeves, arresters, lead joints and crimping bolts; Construct a structured text semantic description for each device, the structured text semantic description including device name, device function, device structural characteristics and spatial relationship between devices; A plurality of scale visual feature maps of each device image and a corresponding structured text semantic description are taken as samples to construct a first training data set ; wherein, denotes a plurality of scale visual feature maps of a first device image; i denotes a plurality of scale visual feature maps of a first device image; denotes a structured text semantic description of a first device image; i denotes a structured text semantic description of a first device image; denotes a total number of samples; Input the multi-scale visual feature maps and the structured text semantic description in the first training data set into a visual projection module and a semantic projection module respectively to generate corresponding visual feature vectors and semantic feature vectors respectively; Based on the generated visual feature vectors and semantic feature vectors, the visual projection module and the semantic projection module are trained end to end according to a first loss function to obtain trained visual projection module and semantic projection module.

7. The valve hall equipment identification method based on multi-scale feature fusion and semantic matching according to claim 6, characterized in that, The first loss function is an adaptive dynamic contrast learning loss function, and the formula is: wherein, is a first loss function; denotes a visual feature vector generated by a visual projection module for the i-th device image; i denotes a visual feature vector generated by a visual projection module for the i-th device image; denotes a semantic feature vector generated by a semantic projection module for the i-th device; i denotes a semantic feature vector generated by a semantic projection module for the i-th device; , is a preset weight coefficient; is a cosine similarity; denotes an adaptive dynamic inter-class negative sample weight of a sample in the i-th batch during training; t denotes an adaptive dynamic inter-class negative sample weight of a sample in the i-th batch during training; i denotes an adaptive dynamic inter-class negative sample weight of a sample in the i-th batch during training; j denotes an adaptive dynamic inter-class negative sample weight of a sample in the i-th batch during training; denotes a current training batch number.​ 8. The valve hall equipment identification method based on multi-scale feature fusion and semantic matching according to claim 7, characterized in that, The adaptive dynamic inter-class negative sample weight, the formula is: wherein, is an adaptive weight adjustment coefficient, is a historical average value, representing a historical average value of cosine similarity between the visual feature vector of the sample and the semantic feature vector of the sample i up to the j th batch.

9. The valve hall equipment identification method based on multi-scale feature fusion and semantic matching according to claim 5, characterized in that, Based on the trained visual projection module, the device class information generated by the region recognition module is used to train the positioning module of the device recognition model for a segmentation task to obtain a trained positioning module, specifically including: Valve hall images are collected in a far, middle and near multi-scale manner, and the collected valve hall images are screened, including removing images with serious blur, serious occlusion or abnormal illumination; For the screened valve hall image, through manual labeling, the real boundary box of each valve hall image is obtained ; wherein, represents the upper left corner coordinates of the boundary box; represents the width and height of the boundary box; For the screened valve hall image, a multi-scale feature extraction network is used to obtain a multi-scale visual feature map of the valve hall image, and the multi-scale visual feature map is input into a trained visual projection module to obtain a corresponding visual feature vector ; The visual feature vector is input into a region recognition module, the region recognition module calculates cosine similarity of the visual feature vector and each semantic feature vector in the semantic knowledge base, and a device category with the highest cosine similarity is taken as the device category information of each valve hall image ; combining the visual feature vectors of the individual valve hall images with the equipment class information to form the localization module input features ; The input features are with the real boundary box as a sample, a second training data set is constructed ; wherein, indicates the total number of samples of the second training data set; The second training data set The input features in the second training data set The input positioning module is trained for the segmentation task, and a prediction bounding box is output. According to the prediction bounding box and the real bounding box, the positioning module is trained through a second loss function, and a trained positioning module is obtained.

10. The valve hall equipment identification method based on multi-scale feature fusion and semantic matching according to claim 9, characterized in that, The second loss function, the formula is: in, For the second loss function, The first number predicted by the positioning module is... i Predicted bounding boxes for each valve chamber image; Indicates the first i The intersection-union ratio (IoU) of the predicted bounding boxes and the ground truth bounding boxes for each sample.

Images