A method for identifying the state of a protection device pressure plate through multimodal fusion and an electronic device

Abstract

The present invention relates to a multimodal fusion protection device pressure plate state recognition method and electronic equipment, and belongs to the field of power system relay protection detection technology. In order to solve the detection problem of non-standardized hard pressure plates in the complex environment of substations, multimodal sensors such as visible light, near-infrared, and three-dimensional structured light are integrated to synchronously collect images, thermal imaging, and point cloud data of the pressure plate area. After pre-processing, visual, thermal, and geometric features are extracted using networks such as ResNet and PointNet++. Multi-source information fusion is achieved through the Transformer cross-modal attention mechanism, and state classification and logic verification are performed in combination with industry rules for pressure plate types. Compared with the existing technology, the present invention has the advantages of high detection accuracy under complex conditions such as uneven lighting and oil obstruction.

Description

Technical Field

[0001] The present invention relates to the technical fields of image recognition, multimodal data fusion and power system, and in particular to a multimodal fusion protection device pressure plate state recognition method and electronic equipment. Background Art

[0002] The relay protection pressure plate is a critical component of the relay protection device in the power system. It is primarily used to control the activation and deactivation of the protection device, ensuring that when a fault occurs in the power system, the protection device can operate quickly and accurately, remove the faulty part, and ensure the safe and stable operation of the power grid. Relay protection pressure plates are divided into two types: soft pressure plates and hard pressure plates. The hard pressure plate is a hardware device such as a connector installed on the protection cabinet surface, which serves as a bridge and link between the protection device and the external secondary circuit. The soft pressure plate activates and deactivates a certain function of the protection device's software system, which can be achieved by modifying the software control word of the protection device. The soft and hard pressure plates of the relay protection device have an "AND" relationship. They must be set to "1" at the same time for the protection function to be activated normally. Therefore, the current status of the on-site relay protection soft and hard pressure plates is an important basis for determining whether the relay protection is working properly.

[0003] Currently, high-voltage substations are equipped with relay protection information collection devices that can remotely read information such as soft pressure plates. However, most substations 110kV and below do not have such systems. Operations and maintenance personnel need to be familiar with different relay protection manufacturers and types, and the method of obtaining soft pressure plates basically relies on manual access through the panel. This method requires relatively high technical skills from operations and maintenance personnel, who need to be proficient in the operation of various types of protection devices. Moreover, item-by-item verification is prone to omissions and errors, making pressure plate inspection time-consuming and labor-intensive, and work efficiency is low. At present, there is no better remote monitoring method for hard pressure plates. The system that uses micro switches for detection requires a large investment of funds and manpower costs for installation and maintenance.

[0004] Chinese patent application publication number CN112069902A provides a method and system for identifying pressure plates in substation cabinets, which realizes intelligent recognition of the status of pressure plates in substation cabinets through deep learning model training, wherein the method includes the following steps: image preprocessing, sample annotation and expansion based on the collected inspection images; deep learning model training using the sample set of inspection images to obtain a target detection model for the status of pressure plates in substation cabinets; and performing substation cabinet pressure plate status recognition processing. By processing the inspection images and training the processed images with deep learning models, the intelligent recognition level of power grid inspection images is improved, and the processing efficiency of inspection images is improved. However, most 110kV and below substations have problems such as limited lighting conditions and oil stains blocking the pressure plates, resulting in recognition failures. The above application cannot effectively deal with such scenarios.

[0005] In summary, there is currently a lack of a method and electronic device for identifying the status of a pressure plate of a protection device to solve or partially solve the aforementioned problems. Summary of the Invention

[0006] The purpose of the present invention is to overcome the defects of the above-mentioned existing technologies and provide a multi-modal fusion method for identifying the status of the pressure plate of a protective device. By integrating multi-source data such as visible light, near-infrared, and three-dimensional structured light, and combining deep learning with industry rule reasoning, it can achieve precise positioning, status recognition and dynamic adaptation of the pressure plate in complex scenarios, thereby improving detection accuracy and engineering practicality.

[0007] The purpose of the present invention can be achieved by the following technical solutions:

[0008] One aspect of the present invention provides a multi-modal fusion method for identifying the state of a protective device pressure plate, characterized by comprising the following steps:

[0009] Acquire the synchronously collected visible light data, infrared image data, depth point cloud data of the relay protection device pressure plate, and the measured data of the ambient light intensity;

[0010] Based on the depth point cloud data, geometric parameters of the pressure plate and curvature information of the connected position are extracted through point cloud segmentation and reconstruction, and depth point cloud features at the pressure plate level are constructed;

[0011] Extract multi-scale visible light features based on visible light data, and extract temperature gradient features based on infrared image data;

[0012] Calculating a brightness mean of the image in the visible light data based on the measured light intensity data, and combining the measured light intensity data and the brightness mean to obtain a comprehensive light intensity;

[0013] Based on the comprehensive light intensity, weighting the visible light feature, temperature gradient feature and depth point cloud feature to obtain a visual fusion feature;

[0014] Based on the visual fusion features, a multi-task learning model is used to predict the state of the protective device pressure plate and its confidence level.

[0015] The loss function in the multi-task learning model training process is:

[0016]

[0017]

[0018]

[0019]

[0020]

[0021] in, 、 、 、 They are total loss, multimodal classification loss, multimodal contrast loss, and geometric verification loss, respectively. 、 、 is the weight of multimodal classification loss, multimodal contrast loss, and geometric verification loss, is the parameter, 、 、 They are visible light-infrared, visible light-point cloud, and infrared-point cloud modal contrast losses, represents the exponential function, is the eigenvector cosine similarity operator, 、 Mode 、 Next pair of samples The feature encoding, 、 is the parameter, 、 They are mean square error loss function and cross entropy loss function, respectively. 、 They are respectively the predicted value of the closing angle of the pressure plate connection and the predicted state label, 、 They are the real angle and real state label based on 3D point cloud computing, 、 are the number of samples and the number of modes, respectively. is the true platen state of the th sample, is the modality importance weight calculated by the attention mechanism, For the The modal pair The predicted probability of a sample, is the temperature hyperparameter.

[0022] As a preferred technical solution, in the process of weighting the visible light features, temperature gradient features and depth point cloud features, the weights are implemented using the following formula:

[0023]

[0024]

[0025]

[0026]

[0027]

[0028] in, 、 、 are the weighted weights of visible light features, temperature gradient features and depth point cloud features, respectively. is the comprehensive light intensity, is the preset threshold, 、 、 、 As parameters, 、 are the measured data of light intensity and the mean brightness of visible light image, respectively. 、 are the height and width of the image in the visible light data, Pixels Gray value.

[0029] As a preferred technical solution, it also includes:

[0030] Based on the predicted protection device pressure plate status and its confidence level, combined with the preset pressure plate type industry rules, logic verification is performed to trigger a secondary check for conflicting results;

[0031] Through semi-supervised learning and incremental learning, the pressure plate feature template library is updated using manually labeled data. For newly emerging non-standard pressure plates, temporary templates are generated and the pressure plate feature template library is updated.

[0032] As a preferred technical solution, the industry rules for the pressure plate types include:

[0033] Determine whether the status of the protective pressure plate is consistent with the status of the soft pressure plate. If not, mark it as pending confirmation;

[0034] Determine whether the locking protection function check is triggered when the maintenance pressure plate is put into use. If not, it is determined to be abnormal.

[0035] As a preferred technical solution, for newly emerged non-standard press plates, the process of generating temporary templates and updating the press plate feature template library includes:

[0036] Calculate the Mahalanobis distance for the newly detected pressure plate features and identify new types of pressure plates through adaptive thresholds;

[0037] A teacher-student framework is used to generate pseudo labels, automatically assigning labels to high-confidence unlabeled samples, and manually prioritizing samples with medium confidence.

[0038] The teacher-student model parameters are updated using the elastic weight consolidation algorithm.

[0039] As a preferred technical solution, the multimodal classification loss and multimodal contrast loss are calculated using the following formula:

[0040]

[0041]

[0042] in, is the parameter, is the light intensity, is the preset threshold.

[0043] As a preferred technical solution, the process of weighting the visible light features, temperature gradient features, and depth point cloud features to obtain visual fusion features includes the following steps:

[0044] For the weighted visible light features and temperature gradient features, splicing is performed in the channel dimension;

[0045] For the spliced vectors, weighted deep point cloud features and text data, fusion features are obtained through the fully connected layer.

[0046] As a preferred technical solution, after obtaining visible light data, infrared image data and depth point cloud data, it also includes:

[0047] For visible light data, low-light image enhancement and guided filtering are performed;

[0048] For visible light data and infrared image data, coarse matching based on scale-invariant feature transformation and fine matching based on B-spline surface deformation are performed to achieve image registration;

[0049] For the depth point cloud data, bilateral filtering is performed to remove outliers.

[0050] Another aspect of the present invention provides an electronic device comprising one or more processors, a memory and one or more programs stored in the memory, wherein the one or more programs include instructions for executing the aforementioned multimodal fusion protection device pressure plate state identification method.

[0051] Compared with the prior art, the present invention has at least one of the following beneficial effects:

[0052] (1) Improve recognition accuracy in complex scenes such as uneven lighting and oil occlusion: The present invention solves problems such as uneven lighting and oil occlusion by integrating visible light, infrared, and three-dimensional point cloud data. It combines multimodal classification loss, multimodal contrast loss, and geometric verification loss to jointly train the model, effectively improving recognition in various scenes, especially low-light scenes.

[0053] (2) Combination of industry knowledge and deep learning: The present invention performs logical verification based on the predicted protection device pressure plate status and its confidence level, combined with preset pressure plate type industry rules, triggers secondary verification of conflict results, and uses semi-supervised learning and incremental learning to update the pressure plate feature template library using manually labeled data. For newly emerging non-standard pressure plates, temporary templates are generated and the pressure plate feature template library is updated, realizing type-aware decision-making and rule base verification, thereby reducing misjudgments caused by confusion of pressure plate types.

[0054] (3) Fully consider the influence of lighting conditions on platen state recognition: On the one hand, the present invention introduces three-dimensional point cloud data, and on the other hand, weights the visible light features, temperature gradient features, and depth point cloud features based on the comprehensive light intensity, thereby increasing the weight of the non-visible light modality under low light conditions. Finally, during the model training process, the multimodal classification loss and multimodal contrast loss are weighted based on the light intensity, thereby jointly improving the recognition accuracy under low light conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 Schematic diagram of the process of the multi-modal fusion protection device pressure plate state identification method in the embodiment;

[0056] Figure 2 This is a flowchart of multimodal feature fusion in an embodiment;

[0057] Figure 3 This is a flowchart of the dense area pressure plate detection and segmentation in the embodiment;

[0058] Figure 4 A flowchart of self-supervised dynamic template generation in an embodiment;

[0059] Figure 5 Schematic diagram of an electronic device in an embodiment. DETAILED DESCRIPTION

[0060] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0061] Example 1

[0062] In view of the problems existing in the above-mentioned prior art, this embodiment provides a multi-modal fusion protection device pressure plate state recognition method. This embodiment takes the 220kV substation protection panel cabinet pressure plate detection as an example. Figure 1 , the method comprises the following steps:

[0063] Step S1: synchronously collect multimodal data.

[0064] This embodiment uses a data acquisition device that integrates a visible light camera, a near-infrared camera, a structured light module, a synchronization trigger circuit, and an environmental sensor to collect multimodal data. The visible light camera supports global shutter and automatic white balance, the near-infrared camera has an integrated temperature sensor, and the synchronization trigger circuit is based on an FPGA design, controlling the exposure time difference between the three modal sensors to within 5ms. The environmental sensor integrates a light intensity sensor and a 6-axis IMU.

[0065] Through data collection, point cloud data, visible light images and infrared thermal images with synchronized resolution are obtained.

[0066] Step S2: multimodal data preprocessing.

[0067] (1) Visible light image enhancement:

[0068] Dehazing: The RetinexNet model pre-trained on the COCO dataset is used for dehazing.

[0069] Guided filtering: A guided filter with a kernel size of 5×5 and a regularization parameter of 0.1 is applied to preserve the edge of the platen and suppress noise.

[0070] (2) Infrared image feature extraction:

[0071] Extract the pressure plate area through Otsu-based adaptive threshold segmentation and calculate the temperature mean ,variance and gradient magnitude , and obtain the temperature characteristic map of the 3 channels.

[0072] (3) 3D point cloud processing:

[0073] Bilateral filtering: A bilateral filter with a spatial standard deviation of 0.01m and a range standard deviation of 0.02m is applied to remove high-frequency noise. Outliers are eliminated by using an outlier removal process with k=5 and a standard deviation multiple of 2, thereby reducing the dimensionality of the point cloud density.

[0074] Segmentation: Apply region growing segmentation with a curvature threshold of 0.01 and a minimum number of regional point clouds of 200 to extract the pressure plate point cloud.

[0075] Feature point registration: Coarse registration is performed based on SIFT feature point matching, and fine registration is performed through B-spline surface deformation.

[0076] Step S3: multi-branch feature extraction.

[0077] (1) Visible light branch:

[0078] Protection pressure plates are generally red, green and yellow. The red pressure plate indicates stop and is used for emergency stop or emergency stop. It is often used for power off protection, action protection, etc. The green pressure plate indicates start or start and is used to start equipment or devices. It is often used for overload protection, trip protection, etc. The yellow pressure plate indicates warning or attention and is used for situations that require attention. It is often used for overload protection, grounding protection, etc.

[0079] ResNet50 is used to extract features from five layers, C1-C5, with convolution sampling intervals of 2, 4, 8, 16, and 32, respectively. A SENet channel attention module is added to the C3-C5 layers. Considering that different pressure plate colors correspond to different functions, the weights of key color channels such as red and green are increased by 2 times.

[0080] (2) Infrared branch:

[0081] Construct a 3-layer temperature feature pyramid (16×16 / 32×32 / 64×64), each layer contains 、 and gradient , a total of 9 channel features.

[0082] (3) Point cloud branch:

[0083] With a neighborhood radius of 0.02 m, PointNet++ was used to extract local geometric features, calculate the flatness, tilt angle, and curvature value of the platen, and identify areas with Gaussian curvature greater than 0.001 as edge areas.

[0084] (4) Type branch:

[0085] According to the different positions of the pressure plates connected to the secondary circuit of the protection device, the pressure plates can be roughly divided into two categories: protection function pressure plates and outlet pressure plates. The protection function pressure plates realize certain functions of the protection device, such as the activation and deactivation of main protection, distance protection, zero-sequence protection, etc. The outlet pressure plates determine the results of the protection action. According to the different objects of the protection action outlet, they can be divided into tripping outlet pressure plates and starting pressure plates.

[0086] Considering the representation of the pressure plate type by the characters in the pressure plate label, the pressure plate type text obtained by input or visible light recognition, such as "zero-sequence protection pressure plate", is encoded into a high-dimensional semantic vector, namely, type feature.

[0087] Step S4: multimodal feature fusion.

[0088] 1. Use the Transformer encoding layer with d_model=512 and nhead=8 to perform cross-attention calculation on visible light features and infrared features. The formula is as follows:

[0089]

[0090] in, Q From the visible light feature projection, K 、 V They come from infrared feature projection respectively, and the output fusion feature contains the inter-modal dependency. is the dimension of the key vector.

[0091] 2. The geometric parameters of flatness, tilt angle and curvature value of the connected position are mapped through two layers of MLP and spliced with the image features to obtain the fusion vector.

[0092] Step S5: type-aware state classification.

[0093] Using a multi-task model structure, the fusion features are fully connected through two layers and output in parallel:

[0094] (1) Status classification, specifically including three categories, namely entry, exit, and abnormal.

[0095] (2) Confidence score, using Softmax to output probability, with temperature scaling parameter T=0.8.

[0096] During the multi-task model training process, the model is trained by combining multimodal classification loss, multimodal contrast loss, and geometric verification loss.

[0097] 1. Multimodal classification loss

[0098]

[0099] Where, is the true platen state of the th sample, 、 are the number of samples and the number of modes, respectively. is the modality importance weight calculated by the attention mechanism, For the The modal pair The predicted probability of a sample. By introducing the modal attention weight , automatically select the optimal modal combination in the current scenario, adopt a weighted summation strategy to fuse multi-modal prediction results, and enhance decision robustness.

[0100] 2. Multimodal contrast loss.

[0101]

[0102]

[0103] Where, As parameters, 、 、 They are visible light-infrared, visible light-point cloud, and infrared-point cloud modal contrast losses, represents the exponential function, is the eigenvector cosine similarity operator, 、 Mode 、 Next pair of samples By adopting a variant of triplet contrast loss, we force different modalities to have similar feature representations for the same platen. In addition, we use the temperature hyperparameter Control contrast intensity

[0104] 3. Geometric verification loss.

[0105]

[0106] Where, 、 is the parameter, 、 They are mean square error loss function and cross entropy loss function, respectively. 、 They are respectively the predicted value of the closing angle of the pressure plate connection and the predicted state label, 、 They are the real angle and real state labels based on 3D point cloud computing. By setting the geometric verification loss, the angle regression and state classification are supervised at the same time to form a joint verification of geometry and state. When the angle is abnormal, it can improve Weight

[0107] 4. Comprehensive losses.

[0108]

[0109] Where, 、 、 、 They are total loss, multimodal classification loss, multimodal contrast loss, and geometric verification loss, respectively. 、 、 are the weights of multimodal classification loss, multimodal contrast loss, and geometric verification loss.

[0110] Step S6, dynamic optimization:

[0111] New pressure plate detection, for example, when a new charging and discharging protection pressure plate is detected, the Mahalanobis distance D² = 2.8 > threshold 2.5 is calculated, and it is determined to be a new type. A temporary template is automatically generated and added to the prototype library.

[0112] To verify the effectiveness of the method in this embodiment in complex environments such as low light and oil obstruction, more than 5,000 pressure plate samples were collected from a 220kV substation. These samples included a benchmark dataset from the early stage of operation, a low-light dataset under low-light scenarios at night, and an oil pollution dataset after a period of operation when the panel on which the pressure plate is located was contaminated with oil due to maintenance and other reasons. Experimental verification was carried out under various loss function configurations, and the results are shown in Table 1.

[0113] Table 1 Experimental results under different loss function configurations

[0114] According to the data, by introducing geometric verification loss and multimodal contrast loss, the accuracy of detection in low-light scenes and oil occlusion scenes is effectively improved.

[0115] Example 2

[0116] Based on Example 1, this embodiment provides another multimodal fusion protection device pressure plate state recognition method. For application scenarios with insufficient light intensity and large changes in light intensity, step S4 is improved. For details, see Figure 2 In this embodiment, the multimodal feature fusion is implemented by the following steps:

[0117] Step S401: Obtain light intensity in real time through the integrated light intensity sensor .

[0118] Step S402: Calculate the average brightness of the visible light image :

[0119]

[0120] Where, 、 are the height and width of the image in the visible light data, Pixels Gray value.

[0121] Step S403: Comprehensively analyze the hardware data and the image analysis results to obtain the comprehensive light intensity. :

[0122]

[0123] Where, 、 As a parameter.

[0124] Step S404: Dynamically adjust the modal weight according to the comprehensive light intensity:

[0125]

[0126]

[0127]

[0128] Where, 、 、 are the weighted weights of visible light features, temperature gradient features and depth point cloud features, respectively. is the preset threshold, 、 , are parameters.

[0129] Step S405: multimodal feature fusion:

[0130]

[0131] in, To fusion features, 、 、 、 They are visible light features, temperature gradient features, depth point cloud features and type features.

[0132] Example 3

[0133] Based on the above embodiment, this embodiment makes targeted improvements to step S2 for the scenario where the density of the pressing plates is high and the spacing between the pressing plates is small. For details, see Figure 3 For visible light images and infrared images, the following dense area pressure plate detection and segmentation process is also included:

[0134] Step S201, super-resolution reconstruction: use the trained super-resolution generative adversarial network to improve the resolution of the image, and restore the edges through bicubic interpolation to improve the clarity of details in dense areas.

[0135] Step S202: construct a feature pyramid: use a three-layer feature pyramid to generate 80×80, 40×40, and 20×20 features, and each layer is processed by SENet enhanced channel attention to obtain a multi-scale enhanced platen feature pyramid.

[0136] Step S203, two-stage detection: Based on the pressure plate feature pyramid, the pressure plate detection result is obtained through a pre-trained two-stage global-local attention network.

[0137] Global localization stage: First, the pyramid is flattened into a sequence and position encoding is added. The features after adding position encoding are processed using a six-layer Transformer encoding layer with d_model=1024 and nhead=16. Deformable convolution is combined to enhance the extraction of dense area features, and the anchor box mechanism is used to obtain globally perceived pressure plate candidate regions.

[0138] The localization stage consists of two branches, one focusing on the main platen features and the other focusing on the adjacent boundary features. In the main feature branch, a dilated convolution with a dilation of 2 is used to extract multi-scale contextual information, thereby expanding the receptive field. In the adjacent boundary branch, a 3×3 convolution kernel with a center weight of 0 is used for convolution to enhance the differences between adjacent platen boundaries.

[0139] During the training process of the global-local attention network, based on the contrastive loss supervision training, the Triplet loss function is used to ensure that the feature spacing of the same type of pressure plate in the trained network is less than 0.2, and the spacing of different types is greater than 0.8.

[0140] Step S204, adaptive post-processing: adaptive non-maximum suppression processing, adjusting the IOU threshold of non-maximum values based on the distance between adjacent pressure plate centers, thus completing the dense area pressure plate detection:

[0141]

[0142] in, is the IOU threshold of non-maximum value, is the parameter, The center distance between adjacent pressure plates, in millimeters.

[0143] Example 4

[0144] Based on the above examples, see Figure 4 In this embodiment, in order to solve the problem that there are many types of pressing plates and it is difficult to generate templates for each type of template, based on step S6, self-supervised dynamic template generation is achieved. The steps include:

[0145] Step S601, unsupervised pre-training: Using the SimCLRv2 framework, the benchmark data is pre-trained using random cropping, color dithering, and other data augmentation techniques to generate positive sample pairs. The feature encoder is then optimized using contrastive loss to ensure that the cosine similarity of features between similar pressure plates is greater than 0.9 and that between different pressure plates is less than 0.5.

[0146] Step S602: Clustering and detection: Using the DBSCAN clustering method, calculate the k-nearest distance threshold to obtain multiple initial templates. Calculate the maximum distance between the new platen feature and the template library, and determine it as a new category if it is greater than the threshold.

[0147] Step S603, model update: Based on the confidence obtained in step S5, use the pre-trained teacher model to assign labels to samples with high confidence and add them to the training set, freeze the backbone network, fine-tune the fusion layer parameters, and perform incremental training.

[0148] Example 5

[0149] Based on the above embodiments, this embodiment provides an electronic device, including: one or more processors and a memory, wherein the memory stores one or more programs, and the one or more programs include instructions for executing the BMS control method described in Example 1.

[0150] like Figure 5 As mentioned above, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 The multimodal fusion protection device pressure plate state identification method. Of course, in addition to software implementation, the present invention does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0151] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0152] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0153] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and such modifications or substitutions are intended to be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A multi-modal fusion protection device pressure plate state recognition method, characterized in that: The steps include: Acquire the synchronously collected visible light data, infrared image data, depth point cloud data of the relay protection device pressure plate, and the measured data of the ambient light intensity; Based on the depth point cloud data, geometric parameters of the pressure plate and curvature information of the connected position are extracted through point cloud segmentation and reconstruction, and depth point cloud features at the pressure plate level are constructed; Extract multi-scale visible light features based on visible light data, and extract temperature gradient features based on infrared image data; Calculating a brightness mean of the image in the visible light data based on the measured light intensity data, and combining the measured light intensity data and the brightness mean to obtain a comprehensive light intensity; Based on the comprehensive light intensity, weighting the visible light feature, temperature gradient feature and depth point cloud feature to obtain a visual fusion feature; Based on the visual fusion features, a multi-task learning model is used to predict the state of the protective device pressure plate and its confidence level. The loss function in the multi-task learning model training process is: in, 、 、 、 They are total loss, multimodal classification loss, multimodal contrast loss, and geometric verification loss, respectively. 、 、 is the weight of multimodal classification loss, multimodal contrast loss, and geometric verification loss, is the parameter, 、 、 They are visible light-infrared, visible light-point cloud, and infrared-point cloud modal contrast losses, represents the exponential function, is the eigenvector cosine similarity operator, 、 Mode 、 Next pair of samples The feature encoding, 、 is the parameter, 、 They are mean square error loss function and cross entropy loss function, respectively. 、 They are respectively the predicted value of the closing angle of the pressure plate connection and the predicted state label, 、 They are the real angle and real state label based on 3D point cloud computing, 、 are the number of samples and the number of modes, respectively. is the true platen state of the th sample, is the modality importance weight calculated by the attention mechanism, For the The modal pair The predicted probability of a sample, is the temperature hyperparameter.

2. The multi-modal fusion protection device pressure plate state recognition method according to claim 1 is characterized in that: In the process of weighting the visible light features, temperature gradient features and depth point cloud features, the weights are implemented using the following formula: in, 、 、 are the weighted weights of visible light features, temperature gradient features and depth point cloud features, respectively. is the comprehensive light intensity, is the preset threshold, 、 、 、 is the parameter, 、 are the measured data of light intensity and the mean brightness of visible light image, respectively. 、 are the height and width of the image in the visible light data, Pixels Gray value.

3. The multi-modal fusion protection device pressure plate state recognition method according to claim 1 is characterized in that: Also includes: Based on the predicted protection device pressure plate status and its confidence level, combined with the preset pressure plate type industry rules, logic verification is performed to trigger a secondary check for conflicting results; Through semi-supervised learning and incremental learning, the pressure plate feature template library is updated using manually labeled data. For newly emerging non-standard pressure plates, temporary templates are generated and the pressure plate feature template library is updated.

4. The multi-modal fusion protection device pressure plate state recognition method according to claim 3 is characterized in that: The industry rules for platen types include: Determine whether the status of the protective pressure plate is consistent with the status of the soft pressure plate. If not, mark it as pending confirmation; Determine whether the locking protection function check is triggered when the maintenance pressure plate is put into use. If not, it is determined to be abnormal.

5. The multi-modal fusion protection device pressure plate state recognition method according to claim 3 is characterized in that: For newly emerged non-standard platens, the process of generating temporary templates and updating the platen feature template library includes: Calculate the Mahalanobis distance for the newly detected pressure plate features and identify new types of pressure plates through adaptive thresholds; A teacher-student framework is used to generate pseudo labels, automatically assigning labels to high-confidence unlabeled samples, and manually prioritizing samples with medium confidence. The teacher-student model parameters are updated using the elastic weight consolidation algorithm.

6. The multi-modal fusion protection device pressure plate state recognition method according to claim 1, characterized in that: The multimodal classification loss and multimodal contrast loss are calculated using the following formula: in, is the parameter, is the light intensity, is the preset threshold.

7. The multi-modal fusion protection device pressure plate state recognition method according to claim 1 is characterized in that: The process of weighting the visible light feature, the temperature gradient feature, and the depth point cloud feature to obtain the visual fusion feature includes the following steps: For the weighted visible light features and temperature gradient features, splicing is performed in the channel dimension; For the spliced vectors, weighted deep point cloud features and text data, fusion features are obtained through the fully connected layer.

8. The multi-modal fusion protection device pressure plate state recognition method according to claim 1, characterized in that: After obtaining visible light data, infrared image data, and depth point cloud data, it also includes: For visible light data, low-light image enhancement and guided filtering are performed; For visible light data and infrared image data, coarse matching based on scale-invariant feature transformation and fine matching based on B-spline surface deformation are performed to achieve image registration; For the depth point cloud data, bilateral filtering is performed to remove outliers.

9. An electronic device, characterized in that: The method comprises one or more processors, a memory and one or more programs stored in the memory, wherein the one or more programs include instructions for executing the multimodal fusion protection device pressure plate state identification method as described in any one of claims 1-8.

Images