Flying and hanging object identification method and device of lightweight open-set detection model, and medium

Abstract

The invention discloses a floating and hanging object identification method and device of a lightweight open-set detection model and a medium, and the method comprises the steps: carrying out the feature extraction and calculation of the multi-mode floating and hanging object identification data of a power transmission line through a detection model, and obtaining the identification result of a floating and hanging object on the power transmission line; wherein the identification result comprises regression coordinates, confidence coefficients and categories of the floating and hanging objects; the detection model is obtained by guiding a neural network model to perform parameter updating through a loss function according to a model identification result. According to the floating and hanging object identification method and device of the lightweight open-set detection model and the medium provided by the invention, through fusion classification and regression loss functions, the classification and positioning accuracy of the model is remarkably improved; meanwhile, key features are captured by means of an attention mechanism, feature dimensions are enriched through multi-modal information fusion, the recognition precision and generalization performance of the model are further enhanced, and the problem that regression coordinates, confidence and categories of the floating and hanging objects are difficult to accurately recognize can be solved.

Description

Technical Field

[0001] The present invention relates to the field of computer recognition technology, and in particular to a method, device and medium for identifying hanging objects using a lightweight open set detection model. Background Art

[0002] As a key component of modern intelligent monitoring systems, hanging object recognition plays a vital role in multiple fields. In industrial production environments, it can effectively prevent accidents caused by falling hanging objects. In public safety management, it can promptly detect and address potential safety hazards. In addition, in nature reserves, hanging object recognition helps protect the ecological environment. Therefore, hanging object recognition technology is of great significance for improving safety management and environmental protection capabilities. Current hanging object recognition technology mainly relies on deep learning-based target detection methods, which can be divided into single-stage target detection and two-stage target detection. Single-stage target detection, such as the YOLO series, is known for its high efficiency. It can directly predict the category and location of objects from images, achieving fast and real-time detection, making it very suitable for real-time application scenarios. Two-stage target detection filters image regions that may contain targets and then carefully processes each region to fully learn target features, making it easier to achieve high detection accuracy.

[0003] However, single-stage target detection technology is prone to losing details while improving performance, resulting in poor recognition of small targets, dense targets, and multi-scale targets. In contrast, although two-stage target detection can provide higher detection accuracy, its network structure is relatively complex and involves a large number of parameters and calculations, making the detection speed relatively slow and unsuitable for real-time scenarios. Moreover, there are many types of hanging objects, which are difficult to fully predefine. Therefore, when encountering unlearned categories, the performance of the relevant prediction model will drop significantly. It often mistakenly classifies unknown categories as known categories due to overfitting the training data, resulting in false detection. Therefore, current hanging object recognition technology has difficulty in effectively identifying the coordinates of hanging objects and their related information. Summary of the Invention

[0004] The present invention provides a method, device and medium for identifying hanging objects based on a lightweight open set detection model, so as to solve the problem of difficulty in accurately identifying the regression coordinates, confidence levels and categories of hanging objects.

[0005] To achieve the above objectives, the present application provides a method for identifying hanging objects using a lightweight open set detection model, comprising:

[0006] Acquire multimodal floating object identification data on transmission lines;

[0007] The multimodal hanging object recognition data is subjected to feature extraction and calculation by a detection model to obtain recognition results of the hanging objects on the transmission line; wherein, the recognition results include the regression coordinates, confidence level and category of the hanging objects; the detection model is obtained by guiding the neural network model to perform parameter updates based on the model recognition results through a loss function; the model recognition results are obtained by feature mapping a number of audiovisual feature encodings and shallow modal features calculated by the neural network model based on an attention mechanism; the loss function is composed of a classification loss function and a regression loss function.

[0008] The present invention uses a detection model to extract and calculate features from multimodal data, enabling rapid identification of objects hanging on power lines. The detection model's loss function combines classification and regression loss functions. This design allows the model to simultaneously optimize classification and regression accuracy during training. The classification loss function helps the model accurately distinguish between different types of objects, while the regression loss function ensures accurate prediction of the object's location. The application of an attention mechanism enables the model to dynamically adjust the importance of different features when processing multimodal data, thereby focusing on features that are critical to the recognition results. By acquiring multimodal data for identifying objects hanging on power lines and leveraging audiovisual feature encoding and shallow modal features, the model effectively fuses multi-source information. This fusion strategy provides a more comprehensive and richer feature representation, thereby improving the model's ability to identify objects hanging. Based on the model's recognition results, the loss function guides parameter updates in the neural network model, an iterative optimization process. As training progresses, the model gradually learns an effective mapping from input data to target output, thereby continuously improving recognition accuracy and generalization.

[0009] Compared with the existing technology, the present invention significantly improves the accuracy of the model in classification and positioning by fusing classification and regression loss functions; at the same time, it uses the attention mechanism to accurately capture key features and enriches the feature dimensions through multimodal information fusion, further enhancing the recognition accuracy and generalization performance of the model, thereby solving the problem of difficulty in accurately identifying the regression coordinates, confidence and category of hanging objects.

[0010] As a preferred solution, the model recognition result is obtained by feature mapping several audiovisual feature codes and shallow modal features calculated by the neural network model based on the attention mechanism, specifically:

[0011] According to the neural network model, visual modal data and language modal data of the preset power transmission line are mapped to a preset high-dimensional vector space to obtain a visual feature vector and a language feature vector;

[0012] fusing the visual feature vector and the language feature vector through an attention mechanism to obtain a fused tensor;

[0013] Performing a nonlinear transformation on the fused tensor to obtain a plurality of audio-visual feature codes;

[0014] Correlation calculation and feature mapping are performed on the plurality of audiovisual feature codes and the preset shallow modal features to obtain the model recognition result; wherein the model recognition result includes the regression coordinates, confidence level and category of the target hanging object.

[0015] This preferred solution maps input data into a high-dimensional vector space, capturing richer feature information. Furthermore, the attention mechanism allows the model to dynamically focus on important information when fusing visual and language features. This helps the model focus more closely on key target hanging features when dealing with real-world scenarios with complex backgrounds and noise. Nonlinear transformations of the fused tensor generate more expressive audiovisual feature encodings. Correlation calculations and feature mapping of these encodings with pre-set shallow modal features further enhance the model's recognition performance.

[0016] As a preferred solution, the visual feature vector and the language feature vector are fused through an attention mechanism to obtain a fused tensor, specifically:

[0017] Capturing spatial correlations between different regions in the visual feature vector and capturing semantic correlations between different words in the language feature vector through a self-attention mechanism, thereby obtaining enhanced visual features and enhanced language features respectively;

[0018] Calculating the attention weight of the visual feature vector to the enhanced language feature, and performing weighted fusion on the enhanced language feature according to the calculated weight to obtain a vision-dominated fusion feature;

[0019] Calculating the attention weight of the language feature vector on the enhanced visual feature, and performing weighted fusion on the enhanced visual feature according to the calculated weight to obtain a language-dominated fusion feature;

[0020] The vision-dominated fusion feature and the language-dominated fusion feature are fused to obtain the fusion tensor.

[0021] This preferred solution not only calculates the attention weights of visual features on language features, but also calculates the attention weights of language features on visual features. This bidirectional cross-attention mechanism allows the model to consider the interaction between the two modalities during the fusion process, thereby improving the accuracy and depth of the fusion. Moreover, by calculating the attention weights of visual features on enhanced language features and performing weighted fusion accordingly, a vision-dominated fusion feature can be generated, which helps the model pay more attention to areas rich in visual information. Similarly, by calculating the attention weights of language features on enhanced visual features and performing weighted fusion accordingly, a language-dominated fusion feature can be generated, which helps the model give more weight to areas where semantic information is important.

[0022] As a preferred solution, a nonlinear transformation is performed on the fused tensor to obtain several audio-visual feature codes, specifically:

[0023] Calculating the dot product similarity of the visual features and the language features in the fusion tensor to obtain a similarity matrix;

[0024] Taking the top N visual-linguistic feature pairs with the highest similarity in the similarity matrix to obtain a feature pair set;

[0025] Calculating a weight for each feature pair in the feature pair set by an optimal transmission algorithm to obtain a weight set;

[0026] The visual features and language features in the feature pair set and the corresponding weights in the weight set are combined into independent units to obtain the plurality of audiovisual feature codes.

[0027] This preferred solution effectively captures the correlation between visual and linguistic features by calculating the dot product similarity between the visual and linguistic features in the fusion tensor. Selecting the top N visual-linguistic feature pairs with the highest similarity effectively identifies the most representative feature combinations, reducing redundant information and making subsequent processing more efficient. Using an optimal transfer algorithm to calculate weights for each feature pair ensures that the weight distribution is consistent with the data's inherent characteristics and optimizes subsequent feature fusion or classification tasks.

[0028] As a preferred solution, correlation calculation and feature mapping are performed on the plurality of audiovisual feature codes and the preset shallow modal features to obtain the model recognition result, specifically:

[0029] Using the plurality of audiovisual feature codes as a query vector and using the preset shallow image features as keys and values, performing correlation calculation and feature mapping to obtain a first feature code;

[0030] Using the first feature code as a query vector and the preset shallow language features as keys and values, performing correlation calculation and feature mapping to obtain a second feature code;

[0031] The second feature code is mapped to a preset output space through a multi-layer perceptron to obtain the model recognition result.

[0032] This preferred solution achieves deep fusion of multimodal features by gradually performing correlation calculations and feature mapping between audiovisual feature encoding and shallow image features and shallow language features; this fusion not only retains the key information in the audiovisual feature encoding, but also integrates the complementary information of image and language modalities, thereby improving the richness and accuracy of feature representation.

[0033] As a preferred solution, the second feature code is mapped to a preset output space through a multilayer perceptron to obtain the model recognition result, specifically:

[0034] Mapping the second feature code to a preset bounding box space through the multilayer perceptron to obtain a regression coordinate prediction set of the target hanging object;

[0035] For each of the plurality of audio-visual feature codes, taking the maximum value of the corresponding row in a preset similarity matrix to obtain a confidence prediction set; and taking the category index corresponding to the maximum value in each of the visual-linguistic feature codes to obtain a category label prediction set;

[0036] According to the confidence prediction set, the regression coordinate with the maximum confidence in the regression coordinate prediction set is used as the first regression coordinate of the target hanging object, the confidence corresponding to the first regression coordinate in the confidence prediction set is used as the first confidence of the target hanging object, and the category corresponding to the first regression coordinate in the category label prediction set is used as the first category of the target hanging object;

[0037] The first regression coordinates, the first confidence level, and the first category constitute the model recognition result of the target hanging object.

[0038] In this preferred embodiment, a multilayer perceptron (MLP) is used to map the second feature code to a preset bounding box space. This bounding box space is a multidimensional space, where each dimension represents a parameter of the bounding box. When the MLP receives the second feature code as input, it gradually refines and transforms these features through multiple layers of nonlinear transformations, ultimately outputting a vector that accurately matches the dimensions of the bounding box space. This vector is the predicted set of regression coordinates for the target hanging object, including the predicted bounding box parameters.

[0039] As a preferred solution, the loss function is composed of a classification loss function and a regression loss function, specifically:

[0040] Calculating an output classification score based on the visual modality data and the language modality data;

[0041] Calculate the classification loss of the output classification score and the preset target matrix according to the cross entropy loss function to obtain a classification loss function;

[0042] Calculating L1 loss and GIOU loss according to the first regression coordinates and the actual measured coordinates of the preset transmission line to obtain the regression loss function;

[0043] The loss function is established according to the classification loss function and the regression loss function.

[0044] The loss function of this preferred solution takes into account both the classification task and the regression task. By jointly optimizing these two tasks, the model can learn how to accurately classify and how to accurately regress at the same time during the training process. This multi-task joint optimization method helps to improve the overall performance of the model.

[0045] The present application also provides a floating object recognition device based on a lightweight open set detection model, comprising a data module and a recognition module;

[0046] Wherein, the data module is used to obtain multi-modal hanging object identification data of the transmission line;

[0047] The recognition module is used to extract and calculate features of the multimodal hanging object recognition data through a detection model to obtain an recognition result of the hanging object on the transmission line; wherein the recognition result includes the regression coordinates, confidence level and category of the hanging object; the detection model is obtained by guiding the neural network model to update parameters based on the model recognition result through a loss function; the model recognition result is obtained by feature mapping a number of audio-visual feature encodings and shallow modal features calculated by the neural network model based on an attention mechanism; the loss function is composed of a classification loss function and a regression loss function.

[0048] As a preferred solution, the recognition module includes a mapping unit, a fusion unit, a transformation unit and a calculation unit;

[0049] The mapping unit is configured to map the visual modality data and the language modality data of the preset power transmission line to a preset high-dimensional vector space according to the neural network model to obtain a visual feature vector and a language feature vector;

[0050] The fusion unit is configured to fuse the visual feature vector and the language feature vector through an attention mechanism to obtain a fusion tensor;

[0051] The transformation unit is used to perform nonlinear transformation on the fused tensor to obtain a plurality of audio-visual feature codes;

[0052] The calculation unit is used to perform correlation calculation and feature mapping on the multiple audio-visual feature codes and preset shallow modal features to obtain the model recognition result; wherein the model recognition result includes the regression coordinates, confidence level and category of the target hanging object.

[0053] As a preferred solution, the fusion unit includes an enhancement subunit, a weight subunit, a fusion subunit and a tensor subunit;

[0054] The enhancement subunit is configured to capture the spatial correlation between different regions in the visual feature vector and the semantic correlation between different words in the language feature vector through a self-attention mechanism, thereby obtaining enhanced visual features and enhanced language features respectively.

[0055] The weight subunit is used to calculate the attention weight of the visual feature vector to the enhanced language feature, and perform weighted fusion on the enhanced language feature according to the calculated weight to obtain a vision-dominated fusion feature;

[0056] The fusion subunit is configured to calculate the attention weight of the language feature vector on the enhanced visual feature, and perform weighted fusion on the enhanced visual feature according to the calculated weight to obtain a language-dominated fusion feature;

[0057] The tensor subunit is used to fuse the vision-dominated fusion features and the language-dominated fusion features to obtain the fusion tensor.

[0058] As a preferred solution, the transformation unit includes a matrix subunit, a feature subunit, a calculation subunit and a combination subunit;

[0059] The matrix subunit is used to calculate the dot product similarity of the visual features and the language features in the fusion tensor to obtain a similarity matrix;

[0060] The feature subunit is used to obtain the top N visual-linguistic feature pairs with the highest similarity in the similarity matrix to obtain a feature pair set;

[0061] The calculation subunit is used to calculate a weight for each feature pair in the feature pair set by using an optimal transmission algorithm to obtain a weight set;

[0062] The combining subunit is used to combine the visual features and language features in the feature pair set and the corresponding weights in the weight set into an independent unit to obtain the multiple audiovisual feature codes.

[0063] As a preferred solution, the calculation unit includes a vector subunit, a mapping subunit and an identification subunit;

[0064] The vector subunit is configured to use the plurality of audiovisual feature codes as a query vector and the preset shallow image features as keys and values, perform correlation calculation and feature mapping, and obtain a first feature code;

[0065] The mapping subunit is configured to use the first feature code as a query vector and preset shallow language features as keys and values ​​to perform correlation calculation and feature mapping to obtain a second feature code;

[0066] The recognition subunit is used to map the second feature code to a preset output space through a multi-layer perceptron to obtain the model recognition result.

[0067] As a preferred solution, the identification subunit is specifically:

[0068] Mapping the second feature code to a preset bounding box space through the multilayer perceptron to obtain a regression coordinate prediction set of the target hanging object;

[0069] For each of the plurality of audio-visual feature codes, taking the maximum value of the corresponding row in a preset similarity matrix to obtain a confidence prediction set; and taking the category index corresponding to the maximum value in each of the visual-linguistic feature codes to obtain a category label prediction set;

[0070] According to the confidence prediction set, the regression coordinate with the maximum confidence in the regression coordinate prediction set is used as the first regression coordinate of the target hanging object, the confidence corresponding to the first regression coordinate in the confidence prediction set is used as the first confidence of the target hanging object, and the category corresponding to the first regression coordinate in the category label prediction set is used as the first category of the target hanging object;

[0071] The first regression coordinates, the first confidence level, and the first category constitute the model recognition result of the target hanging object.

[0072] As a preferred solution, the loss function is composed of a classification loss function and a regression loss function, specifically:

[0073] Calculating an output classification score based on the visual modality data and the language modality data;

[0074] Calculate the classification loss of the output classification score and the preset target matrix according to the cross entropy loss function to obtain a classification loss function;

[0075] Calculating L1 loss and GIOU loss according to the first regression coordinates and the actual measured coordinates of the preset transmission line to obtain the regression loss function;

[0076] The loss function is established according to the classification loss function and the regression loss function.

[0077] The present application also provides a storage medium having a computer program stored thereon. The computer program is called and executed by a computer to implement the above-mentioned method for identifying hanging objects using a lightweight open set detection model. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 This is a flow chart of a method for identifying hanging objects using a lightweight open set detection model provided in an embodiment of the present application;

[0079] Figure 2 This is a schematic diagram of the algorithm structure of the feature fusion module provided in the embodiment of the present application;

[0080] Figure 3 This is a schematic diagram of the structure of the feature decoder algorithm provided in an embodiment of the present application;

[0081] Figure 4 This is a schematic diagram of the overall structure of the model provided in the embodiment of the present application;

[0082] Figure 5 It is a structural schematic diagram of a hanging object recognition device of a lightweight open set detection model provided in an embodiment of the present application. DETAILED DESCRIPTION

[0083] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0084] In the description of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Therefore, features defined as "first" and "second" may explicitly or implicitly include one or more of such features. In the description of this application, unless otherwise specified, "several" means two or more.

[0085] The embodiment of the present application provides a method for identifying hanging objects using a lightweight open-set detection model, which aims to achieve accurate identification and positioning of unknown types of hanging objects on high-voltage transmission lines, breaking through the adaptability limitations of traditional closed-set detection models in complex open environments, thereby improving power inspection efficiency, reducing safety hazards, and promoting the intelligent upgrade of power grid operation and maintenance.

[0086] Example 1:

[0087] See also Figure 1 The embodiment of the present application provides a method for identifying hanging objects using a lightweight open set detection model, including S1 to S2. The specific implementation steps are as follows:

[0088] S1. Acquire multi-modal floating object recognition data of transmission lines;

[0089] Step S1 of the embodiment of the present application is specifically as follows:

[0090] Obtain multimodal identification data of hanging objects on transmission lines, including original visual modal data and original language modal data;

[0091] Among them, raw visual modal data refers to images or video materials shot using different spectral bands, which provide a variety of information about scenes or objects; raw language modal data is the description or label of hanging objects automatically generated by machine learning or deep learning models, and raw language modal data can also be obtained by experts or annotators with professional knowledge and experience manually describing and labeling hanging objects.

[0092] S2. Feature extraction and calculation are performed on the multimodal hanging object recognition data through the detection model to obtain the recognition results of the hanging objects on the transmission line; wherein the recognition results include the regression coordinates, confidence and category of the hanging objects; the detection model is obtained by guiding the neural network model to update the parameters based on the model recognition results through the loss function; the model recognition results are obtained by feature mapping the audio-visual feature encodings and shallow modal features calculated by the neural network model based on the attention mechanism; the loss function is composed of a classification loss function and a regression loss function.

[0093] Step S2 of the embodiment of the present application includes S2.1 to S2.6, wherein S2.1 is a process of obtaining a visual feature vector and a language feature vector according to a feature extraction module, S2.2 is a process of obtaining a fusion tensor according to a feature fusion module, S2.3 is a process of obtaining a plurality of audiovisual feature codes according to a language-guided query module, S2.4 is a process of obtaining a model recognition result according to a feature decoding module, S2.5 is a process of establishing a loss function according to a loss function optimization module; and S2.6 is a process of optimizing a neural network model according to the loss function to obtain a detection model and perform hanging object recognition, specifically:

[0094] S2.1. Establish a neural network model; wherein the neural network model includes a feature extraction module, a feature fusion module, a language-guided query module, a feature decoding module and a loss function optimization module.

[0095] Obtain historical modal floating object recognition data for a preset transmission line, including visual modal data and language modal data;

[0096] Based on the visual feature extractor in the feature extraction module, the depthwise separable convolution technology is used to process the visual modality data to generate a visual feature vector with a unified dimensionality specification. Among them, the depthwise convolution in the depthwise separable convolution technology focuses on capturing the spatial features of the image; the point-by-point convolution focuses on cross-channel feature combination to achieve efficient utilization of computing resources.

[0097] Based on the language feature extractor in the feature extraction module, the language modality data is bidirectionally encoded using the BERT architecture to generate language feature vectors with uniform dimensionality. BERT (Bidirectional Encoder Representations from Transformers) is a pre-trained language representation model.

[0098] The visual feature vector and the language feature vector are integrated through the fusion layer.

[0099] The feature extraction module, as the core component of the neural network model, is located at the initial input of the algorithm processing chain. Its main function is to map heterogeneous visual and language modality data into a high-dimensional vector space to extract image and text features. This module consists of two parallel components: a visual feature extractor and a language feature extractor.

[0100] On the one hand, the core component of the visual feature extractor is the depthwise separable convolution technology. This technology decomposes the traditional convolution operation into two parts: depthwise convolution and pointwise convolution. Depthwise convolution focuses on the fine capture of image spatial features, while pointwise convolution is committed to the efficient combination of cross-channel features; this decomposition can not only significantly improve the computational efficiency of the model, but also greatly reduce the number of parameters. Therefore, this technology can achieve the optimal utilization of computing resources, ensuring that the model can also demonstrate powerful real-time processing capabilities on edge devices while maintaining high-precision performance;

[0101] The language feature extractor, on the other hand, is rooted in the BERT architecture. Thanks to its deep pre-training on large-scale text datasets, the BERT architecture has accumulated rich language representation capabilities. Through targeted fine-tuning, BERT can easily handle a variety of specific downstream tasks, demonstrating excellent performance. Particularly noteworthy is BERT's bidirectional encoding feature, which simultaneously considers both previous and subsequent contextual information, providing solid support for accurately capturing and utilizing language information. This feature demonstrates significant advantages in understanding and generating descriptions closely related to images.

[0102] In this embodiment, S2.1, leveraging this dual-stream network architecture within the feature extraction module, it processes visual and language information in parallel and integrates these features through a fusion layer. This approach not only improves processing efficiency but also enhances the model's expressiveness when processing complex multimodal data, making it more accurate in classifying, detecting, or generating descriptions of image content.

[0103] S2.2. Based on the feature fusion module, the self-attention mechanism is used to capture the spatial correlation between different regions in the visual feature vector, such as the correlation between hanging objects and surrounding objects, and to capture the semantic correlation between different words in the language feature vector, such as the relationship between "white" and "hanging objects" in "white hanging objects", to obtain enhanced visual features and enhanced language features respectively.

[0104] The visual feature vector is used as the query vector Q, and the enhanced language feature is used as the key K and value V to calculate the attention weight of the visual feature vector to the enhanced language feature. The enhanced language feature is weightedly fused according to the calculated weight to obtain the vision-dominated fusion feature;

[0105] The language feature vector is used as the query vector Q, and the enhanced visual feature is used as the key K and value V to calculate the attention weight of the language feature vector on the enhanced visual feature. The enhanced visual feature is weightedly fused according to the calculated weight to obtain the language-dominated fusion feature;

[0106] The vision-dominated fusion features and the language-dominated fusion features are fused to obtain a preliminary fusion tensor; the preliminary fusion tensor is nonlinearly transformed through a multi-layer perceptron (MLP) to further integrate cross-modal information and obtain a fusion tensor with a joint spatial-semantic representation.

[0107] The feature fusion module is deployed downstream of the feature extraction module, connecting them in a cascaded fashion. The core of this module is a carefully constructed layered interactive architecture based on a cross-modal attention mechanism. This architecture dynamically associates visual feature vectors with language feature vectors, enabling deep semantic connections between them. This allows the feature fusion module to fully integrate information from different modalities, providing more comprehensive and accurate feature input for subsequent processing or decision-making. Furthermore, this module includes a multi-head cross-attention sublayer.

[0108] To apply this application example, please refer to Figure 2 , Figure 2 This is a schematic diagram of the algorithm structure of the feature fusion module provided in an embodiment of the present application, which shows the algorithm structure based on the feature fusion module, aiming to process the visual feature vector (image feature) and the language feature vector (text feature) to generate a fusion tensor.

[0109] This embodiment S2.2 uses a bidirectional self-attention mechanism, which significantly enhances the model's interactivity and deep understanding capabilities when processing multimodal data.

[0110] S2.3. The language-guided query module uses the visual features in the fused tensor as the query vector Q and the language features in the fused tensor as the key K and value V, calculates the dot product similarity, and generates a similarity matrix of size [N, M], where N is the number of visual features and M is the number of language features.

[0111] All N×M values ​​of the similarity matrix are globally sorted in descending order, and the top 900 visual-linguistic feature pairs with the highest similarity are selected to obtain a feature pair set; each feature pair in the feature pair set consists of a pair of visual features and language features;

[0112] Assigning a weight to each feature pair in the feature pair set through the optimal transfer algorithm to obtain a weight set; and the weight represents the matching confidence of the visual-language feature pair;

[0113] The visual features, language features, and corresponding weights from the weight set for each feature pair in the feature pair set are combined into independent units to obtain several audiovisual feature encodings (i.e., visual-linguistic feature encodings). These audiovisual feature encodings include classification information and localization information. Classification information refers to the category probability distribution calculated through cross-modal contrastive learning, indicating the likelihood that the target belongs to each category. Localization information refers to the bounding box parameters output by the regression head, which are used for subsequent target localization.

[0114] Furthermore, during the training process, the model parameters of the query module are optimized through the following steps:

[0115] A bipartite graph matching problem is constructed between the predicted boxes and the true labels. The matching cost is composed of the weighted sum of the L1 loss (coordinate offset) and the GIOU loss (box overlap), thereby obtaining the total matching cost. Among them, the "prediction box" refers to the rectangular box generated by the model during training to locate the target in the image. These boxes will be matched with the true label to determine the model's prediction accuracy and update the model weights accordingly. In addition, the L1 loss, also known as the absolute error loss or Manhattan distance, is a method to measure the difference between two vectors. The GIOU loss is a loss function that measures the overlap between two bounding boxes. It is an extension of the IOU (Intersection over Union) loss.

[0116] Determine the global optimal matching matrix through the Hungarian algorithm to minimize the total matching cost;

[0117] The classification loss and regression loss are calculated based on the target matching matrix, and the model parameters of the query module are updated through back propagation.

[0118] The language-guided query module, tightly connected to the output of the feature fusion module in a cascaded fashion, constructs a multi-layered, interrelated analysis network based on an advanced cross-modal semantic retrieval framework. The module first accurately calculates the joint probability distribution between pairs of visual and linguistic features using a similarity metric matrix in the feature space. Subsequently, it applies a differentiable ranking operator to optimize feature selection based on the degree of cross-modal matching under the constraints of the probability distribution. Furthermore, the module employs an optimal transfer matching algorithm to cleverly generate a weighted image feature index, providing strong support for subsequent processing and enabling the use of input text to guide object detection.

[0119] In this embodiment S2.3, by calculating the dot product similarity of the visual and linguistic features in the fusion tensor, it is possible to effectively capture the association between visual and linguistic features. Selecting the top 900 visual-linguistic feature pairs with the highest similarity can efficiently screen out the most representative feature combinations. This approach reduces redundant information and makes subsequent processing more efficient. Using an optimal transfer algorithm to calculate weights for each feature pair ensures that the weight distribution is consistent with the characteristics of the data itself and can optimize subsequent feature fusion or classification tasks.

[0120] S2.4. Input the audiovisual feature codes into the decoder of the feature decoding module and undergo a round of self-attention processing. During this process, the elements within the code interact with each other, thereby strengthening the connection and consistency between features.

[0121] The first feature encoding is obtained by encoding the audiovisual features processed by self-attention as the query vector Q, and performing correlation calculation and feature mapping using the preset shallow image features as the key K and value V to capture and enhance the contextual details related to the visual information. The shallow image features are extracted from the visual modality data and represent the basic structure and texture information in the image.

[0122] The first feature code is used as the query vector Q, and the preset shallow language features are used as the key K and value V. Correlation calculation and feature mapping are performed to further deepen the understanding and integration of the language content, thereby obtaining the second feature code. The shallow language features are extracted from the language modality data and represent the basic vocabulary and grammatical information in the text.

[0123] The second feature encoding is mapped to the preset bounding box space through a multi-layer perceptron to obtain the query key, that is, the regression coordinate prediction set of the target hanging object. In addition, the number of query keys is consistent with the number of keys generated by the language-guided query module and is used for subsequent loss function regression and classification tasks; this means that for each feature input to the decoder, the decoder will output a corresponding query key (regression coordinate); the number of keys generated by the language-guided query module determines the number of query keys output in this step. In addition, the second feature encoding can be further processed by the regression head inside the decoder to output a boxes tensor including bounding box parameters, that is, to obtain the regression coordinate prediction set of the target hanging object. Here, the "boxes tensor" refers to the tensor including bounding box parameters.

[0124] For each audio-visual feature code (visual-language feature code) in several audio-visual feature codes, take the maximum value of the corresponding row in the similarity matrix to obtain a confidence prediction set; and take the category index corresponding to the maximum value in each audio-visual feature code in the image feature index to obtain a category label prediction set; wherein, the similarity matrix is ​​calculated by step S2.3.

[0125] According to the confidence prediction set, the regression coordinate with the highest confidence in the regression coordinate prediction set is used as the first regression coordinate of the target hanging object, the confidence corresponding to the first regression coordinate in the confidence prediction set is used as the first confidence of the target hanging object, and the category corresponding to the first regression coordinate in the category label prediction set is used as the first category of the target hanging object;

[0126] The first regression coordinate, the first confidence level, and the first category constitute the model recognition result of the target hanging object.

[0127] The feature decoding module is a multi-stage feature reconstruction network. Its front end receives a number of audiovisual feature codes that have been pre-processed with semantic alignment and are rich in contextual information. These codes are derived from the output of the language-guided query module. The core task of this module is to deeply integrate high-level feature information with shallow image features and shallow language features. This process aims to accurately restore and enhance the subtle position information of the regression coordinates in the shallow features. To achieve this goal, the module adopts an advanced self-attention mechanism and combines it with a two-stage cross-attention query strategy. This fusion mechanism ensures that information can be comprehensively analyzed from multiple dimensions, thereby significantly improving the model's understanding ability and response accuracy.

[0128] To apply this application example, please refer to Figure 3 , Figure 3 This is a schematic diagram of the feature decoder algorithm structure provided in an embodiment of the present application, which shows the algorithm structure of the feature decoding module.

[0129] In this embodiment S2.4, by gradually calculating the correlation and mapping the audiovisual feature encoding with shallow image features and shallow language features, a deep fusion of multimodal features is achieved. This fusion not only retains the key information in the audiovisual feature encoding but also integrates the complementary information of the image and language modalities, thereby improving the richness and accuracy of the feature representation.

[0130] Furthermore, a multilayer perceptron is used to map the second feature code to a preset bounding box space. This bounding box space is a multidimensional space, where each dimension represents a parameter of the bounding box. When the multilayer perceptron receives the second feature code as input, it gradually refines and transforms these features through multiple layers of nonlinear transformations, ultimately outputting a vector that accurately matches the dimensions of the bounding box space. This vector is the predicted set of regression coordinates of the target hanging object, which includes the predicted bounding box parameters.

[0131] In addition, the feature decoder of this embodiment not only optimizes the conversion process from raw input to complex feature representation through a series of hierarchical and complementary processing steps, but also significantly improves the accuracy and generalization ability of the final model output by virtue of a sophisticated attention regulation mechanism.

[0132] S2.5. Based on the loss function optimization module, the output classification score S is calculated according to the image feature O and the language feature P; where S = OP T , the image feature O is obtained by the visual feature extractor E I (·) The input image Img in the visual modality data is processed and the calculation formula is O=E I (Img); the language feature P is obtained by the language feature extractor E L (·) The input prompt word Prompt in the language modality data is processed and the calculation formula is P=E L (Prompt).

[0133] According to the cross entropy loss function, the classification loss of the output classification score S and the target matrix T is calculated to obtain the classification loss function L C ; Moreover, when performing the classification loss function matching calculation, if a phrase matches an image region (positive match), all sub-words of the phrase are marked as positive in the target matrix, and all additional marks are negatively matched with all image features;

[0134] Among them, L C=loss(S,T), where loss(·) is the cross entropy loss function. T is the target matrix reconstructed by the binary Hungarian matching method. In particular, in the model of this embodiment, since the length of the input prompt word is difficult to fix, it may be a sentence or a phrase, and often multiple words represent a type. Therefore, when using the binary Hungarian matching algorithm, it is necessary to convert the original target matrix T∈{0,1} N×c The dimension is expanded to T′∈{0,1} N×M , where M is the length of the input prompt word. Normally, the length of M is greater than the length c of the classification matrix in existing research.

[0135] According to the first regression coordinates and the actual measured coordinates of the preset transmission line, the L1 loss and GIOU loss are calculated to obtain the regression loss function L R Among them, the L1 loss helps to reduce the mean absolute error between the predicted box (the first regression coordinate) and the true box (the actual measured coordinate), while the GIOU loss further considers the intersection-over-union ratio between bounding boxes, as well as their shape and size differences. This makes the model more accurate when processing objects with complex overlaps and different scales, allowing the model to effectively identify and process unseen categories in a wider range of application scenarios, and improves the model's adaptability to novel and changing scenarios.

[0136] According to the classification loss function L C And the regression loss function L R Calculate the loss function L; where L = L R +L C , and the loss function is used to quantify the difference between the model's prediction results and the true value, which is the objective function of the neural network optimization process.

[0137] The loss function in S2.5 of this embodiment considers both the classification and regression tasks. By jointly optimizing these two tasks, the model can simultaneously learn how to accurately classify and precisely regress during training. This multi-task joint optimization approach helps improve the overall performance of the model.

[0138] Moreover, the setting of positive and negative matching ensures that the calculation of the loss function can accurately reflect the degree of matching of the model between the image area and the relevant text phrases.

[0139] S2.6. Use the loss function to guide parameter updates in the neural network model to obtain a detection model. Specifically, through the backpropagation algorithm, the chain rule is applied layer by layer to accurately calculate the gradient of the loss function with respect to the parameters of each layer. This process allows error information to be transmitted from the output layer to the bottom layer of the network. Subsequently, the network parameters of the neural network model are adjusted accordingly based on this gradient information, aiming to gradually reduce the loss value until a high-performance detection model is trained.

[0140] The detection model extracts and calculates features of the multimodal hanging object recognition data to obtain the recognition results of the hanging objects on the transmission line; the recognition results include the regression coordinates, confidence level and category of the hanging objects.

[0141] To apply the embodiments of this application, please refer to Figure 4 , Figure 4 This is a schematic diagram of the overall structure of the model provided in the embodiment of the present application, which shows the algorithm data processing flow of the "feature extraction module, feature fusion module, language-guided query module and feature decoding module" in this embodiment. The following is a general description of the algorithm data processing flow of this embodiment in conjunction with this figure:

[0142] First, in the multimodal feature encoding stage, the input multispectral visual data and the model-annotated description of the hanging object are fed into the visual feature extractor and language feature extractor in the feature extraction module, respectively. These two extractors work in parallel, outputting dimensionally normalized visual and language feature vectors.

[0143] Secondly, the feature fusion module establishes semantic associations between visual and language feature vectors through self-attention and multi-head cross-attention mechanisms, generating a fusion tensor with a joint spatial-semantic representation.

[0144] Next, the fused tensor is sorted by feature significance using a differentiable sorting operator in the query module. An optimal transfer matching algorithm is used to construct a visual-language association matrix and select the top 900 high-response regions to form a cross-modal joint encoding.

[0145] Finally, the feature decoding module receives these joint codes and, through a multi-stage feature reconstruction process using residual connections, ultimately outputs the detection results including the regression coordinates, confidence, and category of the hanging objects.

[0146] In summary, this embodiment S2 maps the input data to a high-dimensional vector space, capturing richer feature information. Furthermore, the attention mechanism allows the model to dynamically focus on important information when fusing visual and linguistic features. This helps the model focus more on key target hanging object features when processing real-world scenes with complex backgrounds and noise. Nonlinear transformations of the fused tensor can generate more expressive audiovisual feature encodings. Correlation calculations and feature mapping of these encodings with pre-set shallow modal features further enhance the model's recognition performance.

[0147] It should be noted that the transmission line involved in this embodiment belongs to the category of high-voltage power transmission lines.

[0148] Overall, this embodiment has the following beneficial effects:

[0149] This application uses a detection model to extract and calculate features from multimodal data, enabling rapid identification of objects hanging on power lines. The detection model's loss function combines classification and regression loss functions. This design allows the model to simultaneously optimize classification and regression accuracy during training. The classification loss function helps the model accurately distinguish between different types of objects, while the regression loss function ensures that the model accurately predicts the object's location. The application of an attention mechanism enables the model to dynamically adjust the importance of different features when processing multimodal data, thereby focusing on features that are critical to the recognition results. By acquiring multimodal data for identifying objects hanging on power lines and leveraging audiovisual feature encoding and shallow modal features, the model effectively fuses multi-source information. This fusion strategy provides a more comprehensive and richer feature representation, thereby improving the model's ability to identify objects hanging on power lines. Based on the model's recognition results, the loss function guides the neural network model's parameter updates, an iterative optimization process. As training progresses, the model gradually learns an effective mapping from input data to target output, thereby continuously improving recognition accuracy and generalization.

[0150] In summary, this application utilizes a large visual model to implement open-set detection, a capability crucial for object recognition because it can identify new types of objects not included in the training set, thereby improving the robustness and adaptability of the system. Furthermore, a lightweight image feature extraction method is designed to ensure efficient algorithm operation on edge devices, enabling rapid response and effective utilization of power and computing resources. Therefore, this application can handle diverse recognition tasks while meeting the real-time requirements of practical applications.

[0151] Example 2:

[0152] See also Figure 5, the embodiment of the present application provides a hanging object recognition device based on a lightweight open set detection model, comprising a data module 10 and a recognition module 20;

[0153] The data module 10 is used to obtain multi-modal hanging object identification data of the transmission line;

[0154] The recognition module 20 is used to extract and calculate features of multimodal hanging object recognition data through a detection model to obtain recognition results of hanging objects on the transmission line; wherein the recognition results include the regression coordinates, confidence level and category of the hanging objects; the detection model is obtained by guiding the neural network model to update parameters based on the model recognition results through a loss function; the model recognition results are obtained by feature mapping of several audio-visual feature encodings and shallow modal features calculated by the neural network model based on the attention mechanism; the loss function is composed of a classification loss function and a regression loss function.

[0155] In one embodiment, the data module 10 is specifically:

[0156] Obtain multimodal identification data of hanging objects on transmission lines, including original visual modal data and original language modal data;

[0157] Among them, raw visual modal data refers to images or video materials shot using different spectral bands, which provide a variety of information about scenes or objects; raw language modal data is the description or label of hanging objects automatically generated by machine learning or deep learning models, and raw language modal data can also be obtained by experts or annotators with professional knowledge and experience manually describing and labeling hanging objects.

[0158] In one embodiment, the recognition module 20 includes a mapping unit, an enhancement subunit, a weight subunit, a fusion subunit, a tensor subunit, a matrix subunit, a feature subunit, a calculation subunit, a combination subunit, a vector subunit, a mapping subunit, an identification subunit, a function unit, and an optimization unit;

[0159] Among them, the mapping unit is the process of obtaining visual feature vectors and language feature vectors according to the feature extraction module; the enhancement subunit, weight subunit, fusion subunit and tensor subunit are the processes of obtaining fusion tensors according to the feature fusion module; the matrix subunit, feature subunit, calculation subunit and combination subunit are the processes of obtaining a number of audio-visual feature codes according to the language-guided query module; the vector subunit, mapping subunit and recognition subunit are the processes of obtaining model recognition results according to the feature decoding module; the function unit is the process of establishing the loss function according to the loss function optimization module; the optimization unit is the process of optimizing the neural network model according to the loss function to obtain the detection model and perform hanging object recognition;

[0160] Among them, the mapping unit is used to establish a neural network model; wherein the neural network model includes a feature extraction module, a feature fusion module, a language-guided query module, a feature decoding module and a loss function optimization module.

[0161] The mapping unit is further used to obtain historical modal hanging object recognition data of a preset transmission line, including visual modal data and language modal data;

[0162] The mapping unit is also used to process the visual modality data using the depthwise separable convolution technology based on the visual feature extractor in the feature extraction module to generate a visual feature vector with a unified dimensionality specification; among them, the depthwise convolution in the depthwise separable convolution technology focuses on capturing the spatial features of the image; the point-by-point convolution focuses on cross-channel feature combination to achieve efficient utilization of computing resources.

[0163] The mapping unit is also used to bidirectionally encode the language modality data using the BERT architecture based on the language feature extractor in the feature extraction module to generate a language feature vector with a uniform dimensionality specification. BERT (Bidirectional Encoder Representations from Transformers) is a pre-trained language representation model.

[0164] The mapping unit is also used to integrate the visual feature vector and the language feature vector through the fusion layer.

[0165] The feature extraction module, as the core component of the neural network model, is located at the initial input of the algorithm processing chain. Its main function is to map heterogeneous visual and language modality data into a high-dimensional vector space to extract image and text features. This module consists of two parallel components: a visual feature extractor and a language feature extractor.

[0166] On the one hand, the core component of the visual feature extractor is the depthwise separable convolution technology. This technology decomposes the traditional convolution operation into two parts: depthwise convolution and pointwise convolution. Depthwise convolution focuses on the fine capture of image spatial features, while pointwise convolution is committed to the efficient combination of cross-channel features; this decomposition can not only significantly improve the computational efficiency of the model, but also greatly reduce the number of parameters. Therefore, this technology can achieve the optimal utilization of computing resources, ensuring that the model can also demonstrate powerful real-time processing capabilities on edge devices while maintaining high-precision performance;

[0167] The language feature extractor, on the other hand, is rooted in the BERT architecture. Thanks to its deep pre-training on large-scale text datasets, the BERT architecture has accumulated rich language representation capabilities. Through targeted fine-tuning, BERT can easily handle a variety of specific downstream tasks, demonstrating excellent performance. Particularly noteworthy is BERT's bidirectional encoding feature, which simultaneously considers both previous and subsequent contextual information, providing solid support for accurately capturing and utilizing language information. This feature demonstrates significant advantages in understanding and generating descriptions closely related to images.

[0168] The mapping unit in this embodiment leverages this dual-stream network architecture within the feature extraction module to process visual and language information in parallel, integrating both features through a fusion layer. This approach not only improves processing efficiency but also enhances the model's expressiveness when processing complex multimodal data, resulting in more accurate classification, detection, and description generation of image content.

[0169] The enhancer unit is used to capture the spatial associations between different regions in the visual feature vector through the self-attention mechanism based on the feature fusion module, such as the association between hanging objects and surrounding objects, and to capture the semantic associations between different words in the language feature vector, such as the relationship between "white" and "hanging objects" in "white hanging objects", to obtain enhanced visual features and enhanced language features respectively.

[0170] The weight sub-unit is used to use the visual feature vector as the query vector Q and the enhanced language feature as the key K and value V to calculate the attention weight of the visual feature vector to the enhanced language feature, and perform weighted fusion of the enhanced language feature according to the calculated weight to obtain the vision-dominated fusion feature;

[0171] The fusion subunit is used to use the language feature vector as the query vector Q and the enhanced visual feature as the key K and value V to calculate the attention weight of the language feature vector on the enhanced visual feature, and perform weighted fusion of the enhanced visual feature according to the calculated weight to obtain the language-dominated fusion feature;

[0172] The tensor quantum unit is used to fuse the vision-dominated fusion features and the language-dominated fusion features to obtain a preliminary fusion tensor; the preliminary fusion tensor is nonlinearly transformed through a multi-layer perceptron (MLP) to further integrate cross-modal information and obtain a fusion tensor with a joint spatial-semantic representation.

[0173] The feature fusion module is deployed downstream of the feature extraction module, connecting them in a cascaded fashion. The core of this module is a carefully constructed layered interactive architecture based on a cross-modal attention mechanism. This architecture dynamically associates visual feature vectors with language feature vectors, enabling deep semantic connections between them. This allows the feature fusion module to fully integrate information from different modalities, providing more comprehensive and accurate feature input for subsequent processing or decision-making. Furthermore, this module includes a multi-head cross-attention sublayer.

[0174] To apply this application example, please refer to Figure 2 , Figure 2 This is a schematic diagram of the algorithm structure of the feature fusion module provided in an embodiment of the present application, which shows the algorithm structure based on the feature fusion module, aiming to process the visual feature vector (image feature) and the language feature vector (text feature) to generate a fusion tensor.

[0175] In this embodiment, the enhancement subunit, weight subunit, fusion subunit, and tensor subunit use a bidirectional self-attention mechanism, which significantly enhances the interactivity and deep understanding ability of the model when processing multimodal data.

[0176] The matrix subunit is used in the language-guided query module. It uses the visual features in the fused tensor as the query vector Q and the language features in the fused tensor as the key K and value V to calculate the dot product similarity and generate a similarity matrix of size [N, M], where N is the number of visual features and M is the number of language features.

[0177] The feature subunit is used to globally sort all N×M values ​​of the similarity matrix in descending order, and select the top 900 visual-linguistic feature pairs with the highest similarity to obtain a feature pair set. Each feature pair in the feature pair set consists of a pair of visual features and language features.

[0178] A computing subunit is configured to assign a weight to each feature pair in the feature pair set by using an optimal transfer algorithm to obtain a weight set; and the weight represents a matching confidence of the visual-linguistic feature pair;

[0179] The combination subunit is used to combine the visual features, language features, and corresponding weights in the weight set of each feature pair in the feature pair set into independent units to obtain multiple audio-visual feature codes (i.e., visual-linguistic feature codes). The multiple audio-visual feature codes include classification information and positioning information. Classification information refers to the category probability distribution calculated through cross-modal contrastive learning, indicating the probability of the target belonging to each category. Positioning information refers to the bounding box parameters output by the regression head, which is used for subsequent target positioning.

[0180] Furthermore, during the training process, the model parameters of the query module are optimized through the following steps:

[0181] A bipartite graph matching problem is constructed between the predicted boxes and the true labels. The matching cost is composed of the weighted sum of the L1 loss (coordinate offset) and the GIOU loss (box overlap), thereby obtaining the total matching cost. Among them, the "prediction box" refers to the rectangular box generated by the model during training to locate the target in the image. These boxes will be matched with the true label to determine the model's prediction accuracy and update the model weights accordingly. In addition, the L1 loss, also known as the absolute error loss or Manhattan distance, is a method to measure the difference between two vectors. The GIOU loss is a loss function that measures the overlap between two bounding boxes. It is an extension of the IOU (Intersection over Union) loss.

[0182] Determine the global optimal matching matrix through the Hungarian algorithm to minimize the total matching cost;

[0183] The classification loss and regression loss are calculated based on the target matching matrix, and the model parameters of the query module are updated through back propagation.

[0184] The language-guided query module, tightly connected to the output of the feature fusion module in a cascaded fashion, constructs a multi-layered, interrelated analysis network based on an advanced cross-modal semantic retrieval framework. The module first accurately calculates the joint probability distribution between pairs of visual and linguistic features using a similarity metric matrix in the feature space. Subsequently, it applies a differentiable ranking operator to optimize feature selection based on the degree of cross-modal matching under the constraints of the probability distribution. Furthermore, the module employs an optimal transfer matching algorithm to cleverly generate a weighted image feature index, providing strong support for subsequent processing and enabling the use of input text to guide object detection.

[0185] The matrix subunit, feature subunit, calculation subunit, and combination subunit of this embodiment can effectively capture the association between visual and language features by calculating the dot product similarity of visual features and language features in the fusion tensor. By selecting the top 900 visual-language feature pairs with the highest similarity, the most representative feature combinations can be efficiently screened out. This method reduces redundant information and makes subsequent processing more efficient. The optimal transmission algorithm is used to calculate the weights for each feature pair, ensuring that the distribution of weights is consistent with the characteristics of the data itself and can optimize subsequent feature fusion or classification tasks.

[0186] Vector units are used to encode several audiovisual features into the decoder of the feature decoding module and undergo a round of self-attention processing; during this process, the elements within the encoding interact with each other, thereby strengthening the connection and consistency between features;

[0187] The vector subunit is further configured to encode the self-attentioned audiovisual features as a query vector Q, and to perform correlation calculation and feature mapping using the preset shallow image features as the key K and value V, so as to capture and enhance the contextual details related to the visual information, thereby obtaining a first feature encoding. The shallow image features are extracted from the visual modality data and represent the basic structure and texture information in the image.

[0188] A mapping subunit is configured to use the first feature code as a query vector Q and preset shallow language features as keys K and values ​​V to perform correlation calculation and feature mapping, thereby further deepening the understanding and integration of the language content and obtaining a second feature code. The shallow language features are extracted from the language modality data and represent the basic vocabulary and grammatical information in the text.

[0189] The recognition subunit is used to map the second feature encoding to the preset bounding box space through a multi-layer perceptron to obtain the query key, that is, the regression coordinate prediction set of the target hanging object. In addition, the number of query keys is consistent with the number of keys generated by the language-guided query module and is used for subsequent loss function regression and classification tasks; this means that for each feature input to the decoder, the decoder will output a corresponding query key (regression coordinate); the number of keys generated by the language-guided query module determines the number of query keys output in this step. In addition, the second feature encoding can be further processed by the regression head inside the decoder to output a boxes tensor including bounding box parameters, that is, to obtain the regression coordinate prediction set of the target hanging object. Here, the "boxes tensor" refers to a tensor including bounding box parameters.

[0190] The recognition subunit is further used to take the maximum value of the corresponding row in the similarity matrix for each audio-visual feature code (visual-language feature code) in a number of audio-visual feature codes to obtain a confidence prediction set; and take the category index corresponding to the maximum value in each audio-visual feature code in the image feature index to obtain a category label prediction set; wherein the similarity matrix is ​​calculated by the matrix subunit.

[0191] The identification subunit is further configured to, based on the confidence prediction set, use the regression coordinate with the highest confidence in the regression coordinate prediction set as the first regression coordinate of the target hanging object, use the confidence corresponding to the first regression coordinate in the confidence prediction set as the first confidence of the target hanging object, and use the category corresponding to the first regression coordinate in the category label prediction set as the first category of the target hanging object;

[0192] The recognition subunit is further configured to form a model recognition result of the target hanging object based on the first regression coordinate, the first confidence level and the first category.

[0193] The feature decoding module is a multi-stage feature reconstruction network. Its front end receives a number of audiovisual feature codes that have been pre-processed with semantic alignment and are rich in contextual information. These codes are derived from the output of the language-guided query module. The core task of this module is to deeply integrate high-level feature information with shallow image features and shallow language features. This process aims to accurately restore and enhance the subtle position information of the regression coordinates in the shallow features. To achieve this goal, the module adopts an advanced self-attention mechanism and combines it with a two-stage cross-attention query strategy. This fusion mechanism ensures that information can be comprehensively analyzed from multiple dimensions, thereby significantly improving the model's understanding ability and response accuracy.

[0194] To apply this application example, please refer to Figure 3 , Figure 3 This is a schematic diagram of the feature decoder algorithm structure provided in an embodiment of the present application, which shows the algorithm structure of the feature decoding module.

[0195] In this embodiment, the vector subunit, mapping subunit, and recognition subunit gradually calculate the correlation and feature mapping between the audiovisual feature encoding and the shallow image features and shallow language features, thereby achieving deep fusion of multimodal features. This fusion not only retains the key information in the audiovisual feature encoding, but also integrates the complementary information of the image and language modalities, thereby improving the richness and accuracy of feature representation.

[0196] Furthermore, a multilayer perceptron is used to map the second feature code to a preset bounding box space. This bounding box space is a multidimensional space, where each dimension represents a parameter of the bounding box. When the multilayer perceptron receives the second feature code as input, it gradually refines and transforms these features through multiple layers of nonlinear transformations, ultimately outputting a vector that accurately matches the dimensions of the bounding box space. This vector is the predicted set of regression coordinates of the target hanging object, which includes the predicted bounding box parameters.

[0197] In addition, the feature decoder of this embodiment not only optimizes the conversion process from raw input to complex feature representation through a series of hierarchical and complementary processing steps, but also significantly improves the accuracy and generalization ability of the final model output by virtue of a sophisticated attention regulation mechanism.

[0198] Function unit, used to calculate the output classification score S based on the image feature O and language feature P based on the loss function optimization module; where S = OP T , the image feature O is obtained by the visual feature extractor E I (·) The input image Img in the visual modality data is processed and the calculation formula is O=E I (Img); the language feature P is obtained by the language feature extractor E L(·) The input prompt word Prompt in the language modality data is processed and the calculation formula is P=E L (Prompt).

[0199] The function unit is also used to calculate the classification loss of the output classification score S and the target matrix T according to the cross entropy loss function to obtain the classification loss function L C ; Moreover, when performing the classification loss function matching calculation, if a phrase matches an image region (positive match), all sub-words of the phrase are marked as positive in the target matrix, and all additional marks are negatively matched with all image features;

[0200] Among them, L C =loss(S,T), where loss(·) is the cross entropy loss function. T is the target matrix reconstructed by the binary Hungarian matching method. In particular, in the model of this embodiment, since the length of the input prompt word is difficult to fix, it may be a sentence or a phrase, and often multiple words represent a type. Therefore, when using the binary Hungarian matching algorithm, it is necessary to convert the original target matrix T∈{0,1} N×c The dimension is expanded to T′∈{0,1} N×M , where M is the length of the input prompt word. Normally, the length of M is greater than the length c of the classification matrix in existing research.

[0201] The function unit is also used to calculate the L1 loss and GIOU loss according to the first regression coordinate and the actual measurement coordinate of the preset transmission line to obtain the regression loss function L R Among them, the L1 loss helps to reduce the mean absolute error between the predicted box (the first regression coordinate) and the true box (the actual measured coordinate), while the GIOU loss further considers the intersection-over-union ratio between bounding boxes, as well as their shape and size differences. This makes the model more accurate when processing objects with complex overlaps and different scales, allowing the model to effectively identify and process unseen categories in a wider range of application scenarios, and improves the model's adaptability to novel and changing scenarios.

[0202] Function unit, also used according to the classification loss function L C And the regression loss function L R Calculate the loss function L; where L = L R +L C , and the loss function is used to quantify the difference between the model's prediction results and the true value, which is the objective function of the neural network optimization process.

[0203] The loss function of the function unit in this embodiment takes into account both the classification task and the regression task. By jointly optimizing these two tasks, the model can simultaneously learn how to accurately classify and how to accurately regress during the training process. This multi-task joint optimization method helps improve the overall performance of the model.

[0204] Moreover, the setting of positive and negative matching ensures that the calculation of the loss function can accurately reflect the degree of matching of the model between the image area and the relevant text phrases.

[0205] The optimization unit uses a loss function to guide parameter updates in the neural network model, ultimately producing a detection model. Specifically, the backpropagation algorithm employs the chain rule to accurately calculate the gradient of the loss function with respect to each layer's parameters. This process allows error information to be passed from the output layer to the underlying network layer. The network parameters of the neural network model are then adjusted based on this gradient information, aiming to gradually reduce the loss until a high-performance detection model is trained.

[0206] The optimization unit is also used to extract and calculate features of multimodal hanging object recognition data through the detection model to obtain the recognition results of hanging objects on the transmission line; wherein the recognition results include the regression coordinates, confidence and category of the hanging objects.

[0207] To apply the embodiments of this application, please refer to Figure 4 , Figure 4 This is a schematic diagram of the overall structure of the model provided in the embodiment of the present application, which shows the algorithm data processing flow of the "feature extraction module, feature fusion module, language-guided query module and feature decoding module" in this embodiment. The following is a general description of the algorithm data processing flow of this embodiment in conjunction with this figure:

[0208] First, in the multimodal feature encoding stage, the input multispectral visual data and the model-annotated description of the hanging object are fed into the visual feature extractor and language feature extractor in the feature extraction module, respectively. These two extractors work in parallel, outputting dimensionally normalized visual and language feature vectors.

[0209] Secondly, the feature fusion module establishes semantic associations between visual and language feature vectors through self-attention and multi-head cross-attention mechanisms, generating a fusion tensor with a joint spatial-semantic representation.

[0210] Next, the fused tensor is sorted by feature significance using a differentiable sorting operator in the query module. An optimal transfer matching algorithm is used to construct a visual-language association matrix and select the top 900 high-response regions to form a cross-modal joint encoding.

[0211] Finally, the feature decoding module receives these joint codes and, through a multi-stage feature reconstruction process using residual connections, ultimately outputs the detection results including the regression coordinates, confidence, and category of the hanging objects.

[0212] In summary, the recognition module 20 of this embodiment maps the input data into a high-dimensional vector space, which can capture richer feature information. Furthermore, the attention mechanism allows the model to dynamically focus on important information when fusing visual and language features. This helps the model focus more on key target hanging object features when processing real-world scenes with complex backgrounds and noise. Nonlinear transformation of the fused tensor can generate more expressive audiovisual feature encodings. Correlation calculation and feature mapping of these encodings with pre-set shallow modal features further enhance the model's recognition performance.

[0213] It should be noted that the transmission line involved in this embodiment belongs to the category of high-voltage power transmission lines.

[0214] Overall, this embodiment has the following beneficial effects:

[0215] This application uses a detection model to extract and calculate features from multimodal data, enabling rapid identification of objects hanging on power lines. The detection model's loss function combines classification and regression loss functions. This design allows the model to simultaneously optimize classification and regression accuracy during training. The classification loss function helps the model accurately distinguish between different types of objects, while the regression loss function ensures that the model accurately predicts the object's location. The application of an attention mechanism enables the model to dynamically adjust the importance of different features when processing multimodal data, thereby focusing on features that are critical to the recognition results. By acquiring multimodal data for identifying objects hanging on power lines and leveraging audiovisual feature encoding and shallow modal features, the model effectively fuses multi-source information. This fusion strategy provides a more comprehensive and richer feature representation, thereby improving the model's ability to identify objects hanging on power lines. Based on the model's recognition results, the loss function guides the neural network model's parameter updates, an iterative optimization process. As training progresses, the model gradually learns an effective mapping from input data to target output, thereby continuously improving recognition accuracy and generalization.

[0216] In summary, this application utilizes a large visual model to implement open-set detection, a capability crucial for object recognition because it can identify new types of objects not included in the training set, thereby improving the robustness and adaptability of the system. Furthermore, a lightweight image feature extraction method is designed to ensure efficient algorithm operation on edge devices, enabling rapid response and effective utilization of power and computing resources. Therefore, this application can handle diverse recognition tasks while meeting the real-time requirements of practical applications.

[0217] Example 3:

[0218] An embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the method for identifying hanging objects using a lightweight open set detection model;

[0219] Wherein, the method for identifying hanging objects of a lightweight open set detection model, if implemented in the form of a software functional unit and used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device that can carry the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0220] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for identifying hanging objects based on a lightweight open set detection model, characterized in that: include: Acquire multimodal floating object identification data on transmission lines; Extracting and calculating features of the multimodal hanging object recognition data using a detection model to obtain an identification result of the hanging object on the transmission line; wherein the identification result includes the regression coordinates, confidence level, and category of the hanging object; The detection model is obtained by guiding the neural network model to update parameters based on the model recognition result through the loss function; the model recognition result is obtained by feature mapping several audio-visual feature encodings and shallow modal features calculated by the neural network model based on the attention mechanism; the loss function is composed of a classification loss function and a regression loss function.

2. The method for identifying hanging objects using a lightweight open set detection model according to claim 1, wherein: The model recognition result is obtained by feature mapping several audiovisual feature codes and shallow modal features calculated by the neural network model based on the attention mechanism, specifically: According to the neural network model, visual modal data and language modal data of the preset power transmission line are mapped to a preset high-dimensional vector space to obtain a visual feature vector and a language feature vector; fusing the visual feature vector and the language feature vector through an attention mechanism to obtain a fused tensor; Performing a nonlinear transformation on the fused tensor to obtain a plurality of audio-visual feature codes; Correlation calculation and feature mapping are performed on the plurality of audiovisual feature codes and the preset shallow modal features to obtain the model recognition result; wherein the model recognition result includes the regression coordinates, confidence level and category of the target hanging object.

3. The method for identifying hanging objects using a lightweight open set detection model according to claim 2, wherein: The visual feature vector and the language feature vector are fused through the attention mechanism to obtain a fusion tensor, specifically: Capturing spatial correlations between different regions in the visual feature vector and capturing semantic correlations between different words in the language feature vector through a self-attention mechanism, thereby obtaining enhanced visual features and enhanced language features respectively; Calculating the attention weight of the visual feature vector to the enhanced language feature, and performing weighted fusion on the enhanced language feature according to the calculated weight to obtain a vision-dominated fusion feature; Calculating the attention weight of the language feature vector on the enhanced visual feature, and performing weighted fusion on the enhanced visual feature according to the calculated weight to obtain a language-dominated fusion feature; The vision-dominated fusion feature and the language-dominated fusion feature are fused to obtain the fusion tensor.

4. The method for identifying hanging objects using a lightweight open set detection model according to claim 2, wherein: The fused tensor is subjected to nonlinear transformation to obtain several audio-visual feature codes, specifically: Calculating the dot product similarity of the visual features and the language features in the fusion tensor to obtain a similarity matrix; Taking the top N visual-linguistic feature pairs with the highest similarity in the similarity matrix to obtain a feature pair set; Calculating a weight for each feature pair in the feature pair set by an optimal transmission algorithm to obtain a weight set; The visual features and language features in the feature pair set and the corresponding weights in the weight set are combined into independent units to obtain the plurality of audiovisual feature codes.

5. The method for identifying hanging objects using a lightweight open set detection model according to claim 2, wherein: Correlation calculation and feature mapping are performed on the plurality of audiovisual feature codes and the preset shallow modal features to obtain the model recognition result, specifically: Using the plurality of audiovisual feature codes as a query vector and using the preset shallow image features as keys and values, performing correlation calculation and feature mapping to obtain a first feature code; Using the first feature code as a query vector and the preset shallow language features as keys and values, performing correlation calculation and feature mapping to obtain a second feature code; The second feature code is mapped to a preset output space through a multi-layer perceptron to obtain the model recognition result.

6. The method for identifying hanging objects using a lightweight open set detection model according to claim 5, wherein: The second feature code is mapped to a preset output space through a multilayer perceptron to obtain the model recognition result, specifically: Mapping the second feature code to a preset bounding box space through the multilayer perceptron to obtain a regression coordinate prediction set of the target hanging object; For each of the plurality of audio-visual feature codes, taking the maximum value of the corresponding row in a preset similarity matrix to obtain a confidence prediction set; and taking the category index corresponding to the maximum value in each of the visual-linguistic feature codes to obtain a category label prediction set; According to the confidence prediction set, the regression coordinate with the maximum confidence in the regression coordinate prediction set is used as the first regression coordinate of the target hanging object, the confidence corresponding to the first regression coordinate in the confidence prediction set is used as the first confidence of the target hanging object, and the category corresponding to the first regression coordinate in the category label prediction set is used as the first category of the target hanging object; The first regression coordinates, the first confidence level, and the first category constitute the model recognition result of the target hanging object.

7. The method for identifying hanging objects using a lightweight open set detection model according to claim 6, wherein: The loss function is composed of a classification loss function and a regression loss function, specifically: Calculating an output classification score based on the visual modality data and the language modality data; Calculate the classification loss of the output classification score and the preset target matrix according to the cross entropy loss function to obtain a classification loss function; Calculating L1 loss and GIOU loss according to the first regression coordinates and the actual measured coordinates of the preset transmission line to obtain the regression loss function; The loss function is established according to the classification loss function and the regression loss function.

8. A device for identifying hanging objects based on a lightweight open set detection model, characterized in that: Including data module and identification module; Wherein, the data module is used to obtain multi-modal hanging object identification data of the transmission line; The recognition module is used to extract and calculate features of the multimodal hanging object recognition data through a detection model to obtain an recognition result of the hanging object on the transmission line; wherein the recognition result includes the regression coordinates, confidence level and category of the hanging object; the detection model is obtained by guiding the neural network model to update parameters based on the model recognition result through a loss function; the model recognition result is obtained by feature mapping a number of audio-visual feature encodings and shallow modal features calculated by the neural network model based on an attention mechanism; the loss function is composed of a classification loss function and a regression loss function.

9. The device for identifying hanging objects using a lightweight open set detection model according to claim 8, wherein: The recognition module includes a mapping unit, a fusion unit, a transformation unit and a calculation unit; The mapping unit is configured to map the visual modality data and the language modality data of the preset power transmission line to a preset high-dimensional vector space according to the neural network model to obtain a visual feature vector and a language feature vector; The fusion unit is configured to fuse the visual feature vector and the language feature vector through an attention mechanism to obtain a fusion tensor; The transformation unit is used to perform nonlinear transformation on the fused tensor to obtain a plurality of audio-visual feature codes; The calculation unit is used to perform correlation calculation and feature mapping on the multiple audio-visual feature codes and preset shallow modal features to obtain the model recognition result; wherein the model recognition result includes the regression coordinates, confidence level and category of the target hanging object.

10. A storage medium, characterized in that: The storage medium stores a computer program, which is called and executed by a computer to implement a method for identifying hanging objects using a lightweight open set detection model as described in any one of claims 1 to 7.

Images