Multi-modal fusion protection device pressing plate state identification method and electronic equipment

Abstract

The invention relates to a multi-mode fusion protection device pressing plate state identification method and electronic equipment, and belongs to the technical field of power system relay protection detection. In order to solve the detection problem of the non-standardized hard pressing plate in the complex environment of the transformer substation, images, thermal imaging and point cloud data of a pressing plate area are synchronously collected by integrating visible light, near-infrared, three-dimensional structured light and other multi-mode sensors, visual, thermal and geometric features are extracted by utilizing ResNet, PointNet + + and other networks after preprocessing, and the non-standardized hard pressing plate is detected. Multi-source information fusion is realized through a Transform cross-modal attention mechanism, and state classification and logic verification are carried out in combination with industry rules of pressing plate types. Compared with the prior art, the method has the advantages of high detection accuracy and the like under complicated conditions of uneven illumination, oil stain shielding and the like.

Description

Technical Field

[0001] The present invention relates to the technical fields of image recognition, multimodal data fusion, and power systems, and in particular, to a method for recognizing the state of protection device pressure plates through multimodal fusion and an electronic device. Background Art

[0002] Relay protection pressure plates are important components in power system relay protection devices, mainly used to control the input and output of protection devices, ensuring that in the event of a power system failure, the protection device can quickly and accurately operate to cut off the faulty part and ensure the safe and stable operation of the power grid. Relay protection pressure plates are divided into two types: soft pressure plates and hard pressure plates. Hard pressure plates are hardware devices such as connecting pieces installed on the protection cabinet surface and are the bridge and link between the protection device and the external secondary circuit. Soft pressure plates are the input and output of a certain function of the protection device software system and can be achieved by modifying the software control word of the protection device. The soft pressure plates and hard pressure plates of the relay protection device are in an "AND" relationship, and this protection function can only be normally input when both are set to "1". Thus, the current states of the on-site relay protection soft pressure plates and hard pressure plates are important bases for judging whether the relay protection can work properly.

[0003] Currently, relay protection information acquisition devices are installed in high-voltage substations, which can remotely read information such as soft pressure plates. However, most substations of 110 kV and below do not have such a system. Maintenance personnel need to face different relay protection manufacturers and types, and the basic way to obtain soft pressure plates is to manually view them item by item through the panel. This method has relatively high technical requirements for maintenance personnel, who need to be proficient in the operation methods of various types of protection devices, and it is easy to have omissions and errors when checking item by item, making the pressure plate inspection work time-consuming and laborious, with low work efficiency. There is currently no good means for remote monitoring of hard pressure plates. For a system that detects through microswitches, a large amount of capital and labor costs are required for installation and maintenance.

[0004] Chinese Patent Application Publication No. CN112069902A provides a method and system for recognizing the pressure plates of substation switch cabinets, which realizes the intelligent recognition of the state of substation switch cabinet pressure plates through deep learning model training. The method includes the following steps: performing image preprocessing, sample annotation, and expansion based on the collected inspection images; using the sample set of inspection images to train a deep learning model to obtain a target detection model for the state of substation switch cabinet pressure plates; performing recognition processing on the state of substation switch cabinet pressure plates. By processing the inspection images and training the processed images with a deep learning model, the intelligent recognition level of power grid inspection images is improved, and the processing efficiency of inspection images is increased. However, in most substations of 110 kV and below, there are problems such as limited lighting conditions and oil stains covering the pressure plates, resulting in recognition failures. The above application cannot effectively handle such scenarios.

[0005] In summary, there is currently a lack of a method for identifying the status of the protection device pressure plate and an electronic device to solve or partially solve the aforementioned problems. Summary of the Invention

[0006] The purpose of the present invention is to overcome the defects of the existing technologies described above and provide a method for identifying the status of the protection device pressure plate through multi-modal fusion. By integrating multi-source data such as visible light, near-infrared, and three-dimensional structured light, and combining deep learning with industry rule reasoning, accurate positioning, status identification, and dynamic adaptability of the pressure plate in complex scenarios are achieved, improving the detection accuracy and engineering practicality.

[0007] The purpose of the present invention can be achieved through the following technical solutions: In one aspect of the present invention, a method for identifying the status of the protection device pressure plate through multi-modal fusion is provided, including the following steps: Obtain the visible light data, infrared image data, depth point cloud data of the relay protection device pressure plate collected synchronously, and the measured data of the ambient light intensity; Based on the depth point cloud data, extract the geometric parameters of the pressure plate and the curvature information of the connection piece position through point cloud segmentation and reconstruction, and construct the depth point cloud features at the pressure plate level; Extract multi-scale visible light features based on the visible light data, and extract temperature gradient features based on the infrared image data; Based on the measured data of the light intensity, calculate the brightness mean value of the image in the visible light data, and combine the measured data of the light intensity and the brightness mean value to obtain the comprehensive light intensity; Based on the comprehensive light intensity, weight the visible light features, temperature gradient features, and depth point cloud features to obtain the visual fusion features; Based on the visual fusion features, use a multi-task learning model to predict the status of the protection device pressure plate and its confidence level.

[0008] As a preferred technical solution, during the process of weighting the visible light features, temperature gradient features, and depth point cloud features, the weights are implemented using the following formula:

[0009]

[0010]

[0011]

[0012]

[0013] where 、 、 They are the weighted weights of the visible light feature, the temperature gradient feature, and the depth point cloud feature, respectively. is the comprehensive illumination intensity, is the preset threshold, , , , are parameters, , are the measured data of the illumination intensity and the comprehensive illumination intensity, respectively, , are the height and width of the image in the visible light data, respectively, is the pixel gray value.

[0014] As a preferred technical solution, it further includes: Based on the predicted state of the protection device pressure plate and its confidence level, combined with the preset industry rules of the pressure plate type, perform logical verification, and trigger secondary verification for the conflict result; Through semi-supervised learning and incremental learning, use the manually labeled data to update the pressure plate feature template library. For the newly emerged non-standard pressure plates, generate temporary templates and update the pressure plate feature template library.

[0015] As a preferred technical solution, the industry rules of the pressure plate type include: Judge whether the state of the protection pressure plate is consistent with the state of the soft pressure plate. If not, mark it as to be confirmed; Judge whether the locking protection function verification is triggered when the maintenance pressure plate is put in. If not, determine it as abnormal.

[0016] As a preferred technical solution, the process of generating temporary templates and updating the pressure plate feature template library for the newly emerged non-standard pressure plates includes: Calculate the Mahalanobis distance for the newly detected pressure plate features, and identify the new type of pressure plate through the adaptive threshold; Adopt the teacher-student framework to generate pseudo-labels, automatically assign labels to the unlabeled samples with high confidence, and sort the samples with medium confidence according to the manual annotation priority; Use the elastic weight consolidation algorithm to update the teacher-student model parameters.

[0017] As a preferred technical solution, the loss function in the training process of the multi-task learning model is:

[0018]

[0019]

[0020]

[0021]

[0022] Among them, , , , are the total loss, multi-modal classification loss, multi-modal contrast loss, and geometric verification loss respectively, , , are the weights of the multi-modal classification loss, multi-modal contrast loss, and geometric verification loss, is a parameter, is the modality , is the contrast loss between modalities, , , are the visible light-infrared, visible light-point cloud, and infrared-point cloud modality contrast losses respectively, represents the exponential function, is the cosine similarity operator for feature vectors, , are the modalities , respectively for the sample feature encoding, , are parameters, , are the mean squared error loss function and the cross-entropy loss function respectively, , are the predicted values of the closing angle of the pressure plate connecting piece and the predicted status label respectively, , are the true angle calculated based on 3D point cloud and the true status label respectively, , are the number of samples and the number of modalities respectively, is the true pressure plate status of the i-th sample, is the modality importance weight calculated through the attention mechanism, is the th modality's predicted probability for the th sample, is the temperature hyperparameter.

[0023] As a preferred technical solution, the multi-modal classification loss and the multi-modal contrast loss are calculated using the following formula:

[0024]

[0025] Among them, is a parameter, is the light intensity, is the preset threshold value.

[0026] As a preferred technical solution, the process of weighting the visible light feature, temperature gradient feature, and depth point cloud feature to obtain a visual fusion feature includes the following steps: For the weighted visible light feature and temperature gradient feature, perform splicing in the channel dimension; For the spliced vector, weighted depth point cloud feature, and text data, obtain a fusion feature through a fully connected layer.

[0027] As a preferred technical solution, after obtaining visible light data, infrared image data, and depth point cloud data, it further includes: Perform low-light image enhancement and guided filtering processing on the visible light data; Perform rough matching based on scale-invariant feature transform and fine matching based on B-spline surface deformation on the visible light data and infrared image data to achieve image registration; Perform bilateral filtering processing on the depth point cloud data and remove outliers.

[0028] Another aspect of the present invention provides an electronic device, including one or more processors, a memory, and one or more programs stored in the memory. The one or more programs include instructions for executing the foregoing method for identifying the state of the protection device pressing plate in multi-modal fusion.

[0029] Compared with the prior art, the present invention has at least one of the following beneficial effects: (1) Improve the recognition accuracy in complex scenarios such as uneven illumination and oil stain occlusion: The present invention integrates visible light, infrared, and three-dimensional point cloud data to solve problems such as uneven illumination and oil stain occlusion, and combines multi-modal classification loss, multi-modal contrast loss, and geometric verification loss to jointly train the model, effectively improving the recognition in various scenarios, especially in low-light scenarios.

[0030] (2) Combine industry knowledge with deep learning: The present invention performs logical verification based on the predicted state of the protection device pressing plate and its confidence level, combined with the preset industry rules of the pressing plate type, triggers secondary verification for conflicting results, updates the pressing plate feature template library using manually labeled data through semi-supervised learning and incremental learning, generates a temporary template for newly emerging non-standard pressing plates and updates the pressing plate feature template library, realizes type-aware decision-making and rule library verification, and reduces misjudgments caused by confusion of pressing plate types.

[0031] (3) Fully consider the influence of lighting conditions on the recognition of the platen state: On the one hand, this invention introduces three-dimensional point cloud data, and on the other hand, weights the visible light features, temperature gradient features, and depth point cloud features based on the comprehensive light intensity, increasing the weight of non-visible light modalities under low light conditions. Finally, during the model training process, weights are applied to the multi-modal classification loss and multi-modal contrast loss based on the light intensity to jointly improve the recognition accuracy under low light conditions. Description of the Drawings

[0032] Figure 1 It is a schematic flow chart of the method for recognizing the platen state of the protection device with multi-modal fusion in the embodiment; Figure 2 It is a flow chart of multi-modal feature fusion in the embodiment; Figure 3 It is a flow chart of detecting and segmenting the platen in the dense area in the embodiment; Figure 4 It is a flow chart of self-supervised dynamic template generation in the embodiment; Figure 5 It is a schematic diagram of the electronic device in the embodiment. Detailed Embodiment

[0033] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative work shall fall within the scope of protection of the present invention.

[0034] Embodiment 1 In view of the problems existing in the foregoing prior art, this embodiment provides a method for recognizing the platen state of a protection device with multi-modal fusion. Taking the detection of the platen of a 220 kV substation protection panel cabinet as an example, see Figure 1 , the method includes the following steps: Step S1, synchronous acquisition of multi-modal data.

[0035] In this embodiment, a data comprehensive acquisition device integrating a visible light camera, a near-infrared camera, a structured light module, a synchronous trigger circuit, and an environmental sensor is used to acquire multi-modal data. Among them, the visible light camera supports a global shutter and automatic white balance. The near-infrared camera is integrated with a temperature sensor. The synchronous trigger circuit is designed based on FPGA to control the exposure time difference of the three-modal sensors within 5 ms. The environmental sensor integrates a light intensity sensor and a 6-axis IMU.

[0036] Point cloud data, visible light images with synchronized resolution, and infrared thermal images are obtained through data acquisition.

[0037] Step S2, Multimodal data preprocessing.

[0038] (1) Visible light image enhancement: Defogging processing: Use the RetinexNet model pre-trained on the COCO dataset for defogging processing.

[0039] Guided filtering: Apply guided filtering with a kernel size of 5×5 and a regularization parameter of 0.1 to retain the edge of the pressing plate and suppress noise.

[0040] (2) Infrared image feature extraction: Extract the pressing plate area through adaptive threshold segmentation based on Otsu, and calculate the mean temperature , variance and gradient magnitude , to obtain a 3-channel temperature feature map.

[0041] (3) 3D point cloud processing: Bilateral filtering: Apply bilateral filtering with a spatial standard deviation of 0.01m and a range standard deviation of 0.02m to remove high-frequency noise. By using outlier removal processing with k = 5 and a standard deviation multiple of 2, eliminate outliers, thereby reducing the point cloud density.

[0042] Segmentation: Apply region growing segmentation with a curvature threshold of 0.01 and a minimum number of region point clouds of 200 to extract the pressing plate point cloud.

[0043] Feature point registration: Coarse registration is performed based on SIFT feature point matching, and fine registration is performed through B-spline surface deformation.

[0044] Step S3, Multi-branch feature extraction.

[0045] (1) Visible light branch: The protection pressing plates are generally red, green, and yellow. The red pressing plate indicates stop and is used for emergency stop or emergency shutdown, and is often used for power-off protection, action protection, etc. The green pressing plate indicates start or initiation and is used to start equipment or devices, and is often used for overload protection, trip protection, etc. The yellow pressing plate indicates warning or attention and is used for situations that require attention, and is often used for overloading protection, grounding protection, etc.

[0046] Use ResNet50 to extract a total of 5 layers of features from C1 - C5, and the convolution sampling intervals of each layer are 2, 4, 8, 16, and 32 respectively. Add a SENet channel attention module to the C3 - C5 layers. Considering that different pressing plate colors correspond to different functions, the weights of key color channels such as red and green are increased by 2 times.

[0047] (2) Infrared branch: Construct a 3 - layer temperature feature pyramid (16×16 / 32×32 / 64×64), and each layer contains , and gradient , with a total of 9 - channel features.

[0048] (3)Point cloud branch: Under the configuration of a neighborhood radius of 0.02m, use PointNet++ to extract local geometric features, calculate the flatness, tilt angle, and curvature value of the connecting plate position, and determine the area with Gaussian curvature > 0.001 as the edge area.

[0049] (4)Type branch: According to the different positions of the connecting plate accessing the secondary circuit of the protection device, the connecting plate can be roughly divided into two categories: protection function connecting plates and outlet connecting plates. The protection function connecting plates realize the input and withdrawal of some functions of the protection device, such as main protection, distance protection, zero - sequence protection, etc. The outlet connecting plates determine the result of the protection action. According to the different objects of the protection action outlet, they can be divided into trip - outlet connecting plates and start - up connecting plates.

[0050] Considering the representation of the connecting plate type by the text in the connecting plate label, encode the connecting plate type text obtained by input or visible - light recognition, such as "zero - sequence protection connecting plate", into a high - dimensional semantic vector, that is, the type feature.

[0051] Step S4, multi - modal feature fusion.

[0052] 1. Adopt a Transformer encoding layer with d_model = 512 and nhead = 8 to perform cross - attention calculation on the visible - light feature and the infrared feature. The formula is as follows:

[0053] Among them, Q comes from the projection of the visible - light feature, K , V come from the projection of the infrared feature respectively. The output fusion feature contains the inter - modal dependence relationship, is the dimension of the key vector.

[0054] 2. Map the geometric parameters of flatness, tilt angle, and curvature value of the connecting plate position through two - layer MLP, and splice them with the image feature to obtain a fusion vector.

[0055] Step S5, type - aware state classification.

[0056] Adopt a multi - task model structure. After the fusion feature passes through 2 - layer fully connected layers, it outputs in parallel: (1)State classification, specifically including 3 categories, namely input, withdrawal, and abnormal.

[0057] (2)Confidence score, using the Softmax output probability, with the temperature scaling parameter T = 0.8.

[0058] During the training process of the multi-task model, the model is trained by combining the multi-modal classification loss, multi-modal contrast loss, and geometric verification loss.

[0059] 1. Multi-modal classification loss

[0060] In the formula, is the true platen state of the th sample, , are the number of samples and the number of modalities respectively, is the modality importance weight calculated by the attention mechanism, is the th modality's predicted probability for the th sample. By introducing the modality attention weight , the optimal modality combination in the current scenario is automatically selected, and the weighted summation strategy is used to fuse the multi-modal prediction results to enhance the decision-making robustness.

[0061] 2. Multi-modal contrast loss.

[0062]

[0063]

[0064] In the formula, is a parameter, is the contrast loss between modalities , , where can be , , , and the corresponding , , are the visible light-infrared, visible light-point cloud, and infrared-point cloud modality contrast losses respectively, represents the exponential function, is the cosine similarity operator of the feature vectors, , are the feature encodings of the sample , under modalities . By adopting a variant of the triplet contrast loss, the feature representations of different modalities for the same platen are forced to be similar. Additionally, the contrast intensity is controlled by the temperature hyperparameter . 3. Geometric verification loss.

[0065]

[0066] In the formula, and are parameters, and are the mean square error loss function and the cross-entropy loss function respectively, and are the predicted values of the closing angle of the pressure plate connection piece and the predicted status label respectively, and are the true angle and the true status label based on the 3D point cloud calculation respectively. By setting the geometric verification loss and simultaneously supervising the angle regression and the status classification, a geometric and status joint verification is formed. When the angle is abnormal, it can improve weight 4. Comprehensive loss.

[0067]

[0068] In the formula, and and and are the total loss, the multi-modal classification loss, the multi-modal contrast loss, and the geometric verification loss respectively, and and are the weights of the multi-modal classification loss, the multi-modal contrast loss, and the geometric verification loss.

[0069] Step S6, dynamic optimization: New pressure plate detection. For example, when a new charging and discharging protection pressure plate is detected, the Mahalanobis distance D² = 2.8 > the threshold 2.5, it is determined as a new type, and a temporary template is automatically generated and added to the prototype library.

[0070] To verify the effectiveness of the method of this embodiment in complex environments such as low light and oil stain occlusion, more than 5,000 pressure plate samples measured in a certain 220 kV substation are collected, including the benchmark data set in the initial stage of production, the low light data set in the low light scene at night, and the oil stain data set after the panel where the pressure plate is located is stained with oil due to maintenance and other reasons after a period of production. Experimental verification is carried out under various loss function configurations, and the results are shown in Table 1.

[0071] Table 1 Experimental results under different loss function configurations

[0072] According to the data, by introducing the geometric verification loss and the multi-modal contrast loss, the detection accuracy in the low light scene and the oil stain occlusion scene is effectively improved.

[0073] Embodiment 2 Based on Embodiment 1, this embodiment provides another method for identifying the state of the protection device pressing plate in multi-modal fusion. For application scenarios with insufficient light intensity and large light intensity changes, step S4 is improved. Specifically, refer to Figure 2 , the multi-modal feature fusion in this embodiment is implemented through the following steps: Step S401, obtain the light intensity in real time through the integrated light intensity sensor .

[0074] Step S402, calculate the brightness mean value of the visible light image :

[0075] In the formula, , are the height and width of the image in the visible light data respectively, is the pixel gray value.

[0076] Step S403, combine the hardware data and the image analysis result to obtain the comprehensive light intensity :

[0077] In the formula, , are parameters.

[0078] Step S404, dynamically adjust the modal weights according to the comprehensive light intensity:

[0079]

[0080]

[0081] In the formula, , , are the weighted weights of the visible light feature, the temperature gradient feature and the depth point cloud feature respectively, is the preset threshold, , are parameters.

[0082] Step S405, multi-modal feature fusion:

[0083] Among them, is the fusion feature, , , , They are visible light features, temperature gradient features, depth point cloud features, and type features respectively.

[0084] Embodiment 3 Based on the foregoing embodiments, in this embodiment, for the scenario with a high platen density and a small platen spacing, a targeted improvement is made to step S2. Specifically, refer to Figure 3 , for visible light images and infrared images, the following dense area platen detection and segmentation process is also included: Step S201, super-resolution reconstruction: Use the pre-trained super-resolution generative adversarial network to improve the resolution of the image, and restore the edges through bicubic interpolation to enhance the clarity of details in the dense area.

[0085] Step S202, construct a feature pyramid: Use a three-layer feature pyramid to generate features of 80×80, 40×40, and 20×20. Each layer is processed by SENet to enhance channel attention, and a multi-scale enhanced platen feature pyramid is obtained.

[0086] Step S203, two-stage detection: Based on the platen feature pyramid, use the pre-trained two-stage global-local attention network to obtain the platen detection result.

[0087] Global localization stage: First, flatten the pyramid into a sequence and add positional encoding, and use six layers of Transformer encoding layers with d_model = 1024 and nhead = 16 to process the features after adding positional encoding. Combine deformable convolution to enhance feature extraction in the dense area, and obtain global-aware platen candidate regions through the anchor box mechanism.

[0088] Local localization stage: It includes two branches, which are used to focus on the main features of the platen and the adjacent boundary features respectively. In the main feature branch, dilated convolution with dilation = 2 is used to extract multi-scale context information, thereby expanding the receptive field. In the adjacent boundary branch, convolution processing is performed using a 3×3 convolution kernel with a central weight of 0 to strengthen the difference between adjacent platen boundaries.

[0089] During the training process of the global-local attention network, supervised training is based on contrastive loss, and the Triplet loss function is used to ensure that the distance between features of the same type of platen in the trained network is <0.2, and the distance between different types is >0.8.

[0090] Step S204, adaptive post-processing: Adaptive non-maximum suppression processing, adjust the IOU threshold of non-maximum values based on the distance between the centers of adjacent platens. Thus, the dense area platen detection is completed:

[0091] Among them, is the IOU threshold of non-maximum values, is a parameter, is the distance between the centers of adjacent pressing plates, with the unit of millimeter.

[0092] Embodiment 4 Based on the foregoing embodiments, refer to Figure 4 , in view of the problem that there are many types of pressing plates in this embodiment and it is difficult to generate templates for each type of template, based on step S6, self-supervised dynamic template generation is realized, and the steps include: Step S601, unsupervised pre-training: First, use the SimCLRv2 framework to perform data augmentation processing such as random cropping and color jitter on the benchmark data to obtain positive sample pairs. Use the contrast loss to optimize and train the feature encoder so that the cosine similarity of the same type of pressing plate features is greater than 0.9, and the cosine similarity of different types is less than 0.5.

[0093] Step S602, clustering and detection: Through the DBSCAN clustering method, calculate the k-nearest neighbor distance threshold to obtain multiple initial templates. Calculate the maximum distance between the new pressing plate features and the template library. When it is greater than the threshold, it is determined as a new category.

[0094] Step S603, model update: Based on the confidence obtained in step S5, use the pre-trained teacher model to assign labels to samples with high confidence and add them to the training set. Freeze the backbone network and fine-tune the parameters of the fusion layer for incremental training.

[0095] Embodiment 5 Based on the foregoing embodiments, this embodiment provides an electronic device, including: one or more processors and a memory. The memory stores one or more programs, and the one or more programs include instructions for executing the BMS control method described in Embodiment 1.

[0096] As Figure 5 described, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory. Of course, it may also include other hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the above Figure 1 described method for identifying the state of the protection device pressing plate with multi-modal fusion. Of course, in addition to the software implementation method, the present invention does not exclude other implementation methods, such as logic devices or a combination of software and hardware. That is to say, the execution subject of the following processing flow is not limited to each logic unit, and can also be hardware or a logic device.

[0097] The memory may include non-permanent memory in the form of computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0098] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile discs (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transitory medium that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media, such as modulated data signals and carrier waves.

[0099] As described above, the above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of various equivalent modifications or substitutions, and these modifications or substitutions should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A method for identifying the state of a protection device pressure plate through multimodal fusion, characterized in that, It includes the following steps: Obtain the visible light data, infrared image data, depth point cloud data of the protection device pressure plate collected synchronously, and the measured data of the ambient light intensity; Based on the depth point cloud data, extract the geometric parameters of the pressure plate and the curvature information of the continuous piece position through point cloud segmentation and reconstruction, and construct the depth point cloud feature at the pressure plate level; Extract multi-scale visible light features based on the visible light data, and extract temperature gradient features based on the infrared image data; Based on the measured data of the light intensity, calculate the brightness mean value of the images in the visible light data, and combine the measured data of the light intensity and the brightness mean value to obtain the comprehensive light intensity; Based on the comprehensive light intensity, weight the visible light features, temperature gradient features and depth point cloud features to obtain the visual fusion features; Based on the visual fusion features, use a multi-task learning model to predict the state and confidence of the protection device pressure plate; 2. The method for identifying the state of the protection device pressing plate according to claim 1 for multimodal fusion, characterized in that, In the process of weighting the visible light features, temperature gradient features and depth point cloud features, the weights are implemented by the following formula: ， ， ， ， ， Among them, , , are the weighted weights of the visible light feature, the temperature gradient feature, and the depth point cloud feature respectively, is the comprehensive illumination intensity, is the preset threshold, , , , are parameters, , are the measured data of the illumination intensity and the comprehensive illumination intensity respectively, , are the height and width of the image in the visible light data respectively, is the pixel gray value.

3. A method for identifying the state of a protection device pressing plate in multi-modal fusion according to claim 1, characterized in that, It also includes: Based on the predicted state and confidence of the protection device pressure plate, perform logical verification in combination with the preset industry rules of the pressure plate type, and trigger secondary verification for the conflict results; Through semi-supervised learning and incremental learning, use the manually labeled data to update the pressure plate feature template library, and generate a temporary template for the newly emerged non-standard pressure plate and update the pressure plate feature template library; 4. A method for identifying the state of the pressure plate of a protection device with multimodal fusion according to claim 3, characterized in that, The industry rules of the pressure plate type include: Judge whether the state of the protection pressure plate is consistent with the state of the soft pressure plate. If not, mark it as to be confirmed; Judge whether the locking protection function verification is triggered when the maintenance pressure plate is put into use. If not, it is judged as abnormal; 5. A method for identifying the state of the protection device pressure plate in multi-modal fusion according to claim 3, characterized in that, The process of generating a temporary template for the newly emerged non-standard pressure plate and updating the pressure plate feature template library includes: Calculate the Mahalanobis distance for the newly detected pressure plate features, and identify the new type of pressure plate through an adaptive threshold; Adopt a teacher-student framework to generate pseudo-labels, automatically assign labels to the unlabeled samples with high confidence, and perform manual annotation priority ranking on the samples with medium confidence; Use the elastic weight consolidation algorithm to update the teacher-student model parameters; 6. The method for identifying the state of the protection device pressing plate in multi-modal fusion according to claim 1, wherein The loss function in the training process of the multi-task learning model is: ， ， ， ， ， Among them, , , , are the total loss, multi-modal classification loss, multi-modal contrast loss, and geometric verification loss respectively, , , are the weights of the multi-modal classification loss, multi-modal contrast loss, and geometric verification loss, is a parameter, is the modality , is the contrast loss between modalities, , , are the visible light-infrared, visible light-point cloud, and infrared-point cloud modality contrast losses respectively, represents the exponential function, is the cosine similarity operator for feature vectors, , are the feature encodings of the sample , under modalities respectively, , are parameters, , are the mean squared error loss function and the cross-entropy loss function respectively, , are the predicted values of the closing angle of the pressure plate connecting piece and the predicted status label respectively, , are the true angle and the true status label calculated based on 3D point cloud respectively, , are the number of samples and the number of modalities respectively, is the true pressure plate status of the is the modality importance weight calculated through the attention mechanism, is the th modality's predicted probability for the th sample, is the temperature hyperparameter.

7. A method for identifying the state of a protection device pressing plate in multi-modal fusion according to claim 6, characterized in that, The multi-modal classification loss and multi-modal contrast loss are calculated by the following formula: ， ， wherein, is a parameter, is the light intensity, is the preset threshold value.

8. A method for identifying the state of the pressure plate of a protection device with multimodal fusion according to claim 1, characterized in that, The process of weighting the visible light features, temperature gradient features and depth point cloud features to obtain the visual fusion features includes the following steps: For the weighted visible light features and temperature gradient features, perform splicing in the channel dimension; For the spliced vector, the weighted depth point cloud features and text data, obtain the fusion features through a fully connected layer; 9. A method for identifying the state of the protection device pressing plate in multi-modal fusion according to claim 1, characterized in that, After obtaining the visible light data, infrared image data and depth point cloud data, it also includes: Perform low-light image enhancement and guided filtering processing on the visible light data; Perform rough matching based on scale-invariant feature transform and fine matching based on B-spline surface deformation on the visible light data and infrared image data to achieve image registration; Perform bilateral filtering processing on the depth point cloud data and remove the outliers.

10. An electronic device, characterized in that, Comprising one or more processors, a memory, and one or more programs stored in the memory, the one or more programs including instructions for performing the method for identifying the state of the protection device pressing plate for multimodal fusion as described in any one of claims 1-9.

Images