Power transmission channel scene multi-modal detection method fusing global semantics and local features

Abstract

The invention provides a power transmission channel scene multi-modal detection method fusing global semantics and local features, and relates to the technical field of data processing, and the method comprises the steps: carrying out the preprocessing of a power transmission data set, and obtaining a power transmission scene image and a power transmission scene text; a multi-modal model is constructed, and the multi-modal model comprises an image feature extractor based on a convolutional neural network and a text feature extractor based on a Transform architecture; performing feature extraction on the power transmission scene image based on an image feature extractor to obtain an image feature map; performing feature extraction on the power transmission scene text based on a text feature extractor to obtain a text feature vector; the image feature map and the text feature vector are fused, a detection result is output based on a multi-modal model, and the detection result comprises recognition and positioning information corresponding to the image target and text semantic judgment information. According to the invention, the accuracy is at least improved, the research and judgment dimension is increased, and the false alarm in a single mode is reduced.

Description

Technical Field

[0001] This invention relates to data processing technology, and more particularly to a multimodal detection method for power transmission channel scenarios that integrates global semantics and local features. Background Technology

[0002] With the rapid development of my country's power system and the in-depth advancement of smart grid construction, the operational safety and stability of transmission lines, as key channels for power energy transmission, are receiving increasing attention. Traditional transmission line inspections mainly rely on manual methods, requiring inspection personnel to periodically conduct visual inspections of the lines and surrounding environment on foot or with the aid of vehicles, helicopters, etc. This is not only inefficient and labor-intensive, but also poses significant safety risks in complex terrain or severe weather conditions, making it difficult to meet the demands of modern power grids for high reliability and highly automated operation and maintenance.

[0003] In recent years, with the rapid development of computer vision and artificial intelligence technologies, image-based target detection methods have been widely used in the identification of potential hazards in power transmission corridor scenarios. However, most existing methods are limited to the identification of local targets, focusing only on the feature information of specific regions in the image, lacking an understanding of the overall context and global semantics of the power transmission corridor scene, and especially unable to comprehensively assess targets by combining specific business information.

[0004] Therefore, there is an urgent need for a multimodal detection method that can effectively integrate global semantic understanding and local feature extraction to improve the accuracy and robustness of identifying potential hazards in power transmission channels, reduce unnecessary false detections, and provide technical support for intelligent inspection. Summary of the Invention

[0005] Based on the above problems, this invention is proposed to provide a multimodal detection method for power transmission channel scenarios that integrates global semantics and local features to overcome or at least partially solve the above problems.

[0006] According to one aspect of the present invention, a multimodal detection method for power transmission channel scenarios that integrates global semantics and local features is provided, comprising:

[0007] Data is collected based on multi-source sensing units, and the power transmission datasets of the corresponding power transmission channels are preprocessed to obtain power transmission scene images and power transmission scene text.

[0008] Construct a multimodal model, wherein the multimodal model includes an image feature extractor based on a convolutional neural network and a text feature extractor based on a Transformer architecture;

[0009] Based on the image feature extractor, feature extraction is performed on the power transmission scene image to obtain an image feature map;

[0010] Based on the text feature extractor, feature extraction is performed on the power transmission scenario text to obtain a text feature vector;

[0011] The image feature map and text feature vector are fused, and the detection result is output based on a multimodal model. The detection result includes the recognition and positioning information of the corresponding image target and the text semantic judgment information.

[0012] Optionally, in the method according to the present invention, data is acquired based on a multi-source sensing unit, and the obtained power transmission data set corresponding to the power transmission channel is preprocessed to obtain power transmission scene images and power transmission scene text, including:

[0013] The multi-source sensing units arranged along the power transmission channel are controlled to collect data on the power transmission channel, resulting in a power transmission dataset consisting of original images and original videos.

[0014] Data cleaning is performed on the power transmission dataset to obtain a cleaned dataset.

[0015] Content annotation is performed on the cleaned dataset to obtain power transmission scene images and power transmission scene text.

[0016] Optionally, in the method according to the invention, each multi-source sensing unit arranged along the transmission channel is controlled to acquire data about the transmission channel, resulting in a transmission data set comprising original images and original video, including:

[0017] Identify the multi-source sensing units arranged along the power transmission channel and obtain the sensing range of each multi-source sensing unit.

[0018] Each multi-source sensing unit is controlled to acquire video data based on its sensing range.

[0019] The video acquisition data is decomposed into track data to obtain image track data and audio track data;

[0020] The audio track data is type-identified, and based on the identification results, it is determined that there is an audio segment in the audio track data corresponding to any human voice type, and the audio percentage of the audio segment corresponding to the audio track data is determined.

[0021] If the audio proportion is greater than a preset proportion, each captured image frame that makes up the image track data is determined as the original image; otherwise, the video captured data is determined as the original video.

[0022] The original image and original video corresponding to each multi-source sensing unit are combined to obtain the power transmission dataset.

[0023] Optionally, in the method according to the invention, the method further includes:

[0024] In response to the audio proportion being greater than a preset proportion, the audio track data is divided into nodes at corresponding preset time intervals to obtain the trend determination points;

[0025] In response to any trend determination point being located in an audio segment corresponding to any human voice type, the trend determination point is designated as a trend identifier point, and the audio decibel value corresponding to each trend identifier point is determined based on the audio segment.

[0026] Based on the chronological order, the time interval between trend marker points in adjacent positions is determined. In response to the trend marker points showing a gradually increasing trend based on the time interval and a gradually decreasing trend based on the audio decibel value, the video acquisition data is determined as the original video. Conversely, each acquired image frame that makes up the image track data is determined as the original image.

[0027] Optionally, in the method according to the present invention, the image feature extractor is a target detection model YOLOv13, wherein the image feature extractor comprises:

[0028] The system consists of two convolutional neural network layers (Conv), one DS-C3k2 layer, and the DS-C3k2 layer is followed by one Conv layer, one DS-C3k2 layer, one depthwise separable convolutional layer (DSConv), one A2C2f layer, one DSConv layer, one A2C2f layer, and two upsample layers.

[0029] Optionally, in the method according to the invention, the depth-separable convolutional layer DSConv comprises:

[0030] The system consists of a depthwise convolutional layer (DWConv), a pointwise convolutional layer (PWConv), a batch normalization layer (BatchNorm), and an activation function (SiLU).

[0031] Optionally, in the method according to the present invention, the DS-C3k2 layer is composed of multiple DS-C3k modules, and the whole is constructed based on the depth-separable convolutional layer DSConv.

[0032] Optionally, in the method according to the present invention, the text feature extractor is composed of a text processing model BERT structure, and the text feature extractor includes: a vector embedding module, a Transformer encoder, and a text output unit;

[0033] The vector embedding module is configured to encode the power transmission scenario text after it has been segmented into sub-words by the word segmenter to obtain the input embedding vector;

[0034] The Transformer encoder includes a self-attention mechanism and a feedforward neural network. The self-attention mechanism is configured to capture dependencies between text sequences, and the feedforward neural network is configured to perform feature transformation on the output of the self-attention mechanism.

[0035] The text output unit is configured to output the probability distribution of randomly masked subwords in the input sequence, and the continuous binary classification probability distribution between two sentences.

[0036] Optionally, in the method according to the present invention, the image feature map and the text feature vector are fused, and the detection result is output based on a multimodal model, including:

[0037] The image feature map and the text feature vector are concatenated and fused, and the fused content is input into the Transformer layer of the multimodal model for feature transformation to obtain the detection result.

[0038] According to another aspect of the present invention, a multimodal detection system for power transmission channel scenarios that integrates global semantics and local features is provided, comprising:

[0039] The data acquisition module is configured to acquire data based on a multi-source sensing unit and preprocess the obtained power transmission data set of the corresponding power transmission channel to obtain power transmission scene images and power transmission scene text.

[0040] The model building module is configured to build a multimodal model, wherein the multimodal model includes an image feature extractor based on a convolutional neural network and a text feature extractor based on a Transformer architecture;

[0041] The image extraction module is configured to extract features from the power transmission scene image based on the image feature extractor to obtain an image feature map;

[0042] The text extraction module is configured to extract features from the power transmission scenario text based on a text feature extractor to obtain a text feature vector;

[0043] The feature fusion module is configured to fuse the image feature map and the text feature vector, and output the detection result based on the multimodal model. The detection result includes the recognition and positioning information of the corresponding image target and the text semantic judgment information.

[0044] According to the present invention, YOLOv13 is used as the backbone network for image feature extraction. While ensuring high-precision target detection capabilities, the lightweight design of the model is fully considered. Specifically, YOLOv13 further optimizes the feature pyramid and attention mechanism in its structure, enabling efficient capture of local detail features at multiple scales. It is particularly suitable for accurate identification of small and occluded targets in complex scenarios such as power transmission channels. Its lightweight architecture significantly reduces computational overhead, making the image feature extraction and inference process faster, meeting the dual requirements of real-time performance and deployment efficiency in practical engineering. Furthermore, in terms of text feature extraction, the present invention introduces the BERT model, fully utilizing its bidirectional Transformer encoding structure to achieve deep semantic understanding of the input text. Unlike traditional methods that rely solely on word order or local windows, BERT can simultaneously consider the dependencies between each word and its surrounding context, and integrate global semantic information of the entire text. By collaboratively utilizing image features extracted by YOLOv13 and text semantic features generated by BERT, it achieves deep fusion of cross-modal information. This framework not only overcomes the inherent defects of traditional pure visual detection models that focus only on local aspects and ignore semantics, but also effectively alleviates the practical problem of technically correct detection results that are irrelevant to business applications. In typical application scenarios such as intelligent inspection of power transmission channels, this invention can combine the equipment status in the image with the operation and maintenance context in the text, significantly improving the accuracy, interpretability, and business relevance of the detection results, providing reliable technical support for the intelligent operation and maintenance of power systems. Attached Figure Description

[0045] Figure 1 A flowchart of a multimodal detection method for power transmission channel scenarios that integrates global semantics and local features according to an embodiment of the present invention is shown;

[0046] Figure 2 This diagram shows a schematic of the backbone of the corresponding target detection model YOLOv13 in this embodiment;

[0047] Figure 3 The structure diagram of DS-C3k2 in YOLOv13;

[0048] Figure 4 This is the structure diagram of A2C2f in YOLOv13;

[0049] Figure 5 The structural diagram of the BERT bidirectional Transformer in this embodiment is shown;

[0050] Figure 6 The flowchart in this embodiment, combined with a practical application scenario, is shown;

[0051] Figure 7A structural block diagram of a multimodal detection system for power transmission channel scenarios that integrates global semantics and local features according to another embodiment of the present invention is shown. Detailed Implementation

[0052] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0053] To address the problems existing in the prior art, the inventors proposed the solution of this invention. One embodiment of this invention provides a multimodal detection method for power transmission channel scenarios that integrates global semantics and local features. This method can be executed in a computing device, which can be understood as a terminal with data processing capabilities, such as a mobile phone or computer.

[0054] Figure 1 A flowchart of a multimodal detection method for power transmission channel scenarios that integrates global semantics and local features according to an embodiment of the present invention is shown, such as... Figure 1 As shown, the multimodal detection method for power transmission channel scenarios that integrates global semantics and local features proposed in this embodiment includes S1-S5, which are detailed below:

[0055] S1. Data is collected based on multi-source sensing units, and the power transmission data of the corresponding power transmission channel is preprocessed to obtain power transmission scene images and power transmission scene text.

[0056] S2. Construct a multimodal model, wherein the multimodal model includes an image feature extractor based on a convolutional neural network and a text feature extractor based on a Transformer architecture;

[0057] S3. Extract features from the power transmission scene image based on the image feature extractor to obtain an image feature map;

[0058] S4. Extract features from the power transmission scenario text using a text feature extractor to obtain a text feature vector;

[0059] S5. The image feature map and text feature vector are fused together, and the detection result is output based on the multimodal model. The detection result includes the recognition and positioning information of the corresponding image target and the text semantic judgment information.

[0060] For example, in this embodiment, firstly, data can be collected based on multi-source sensing units deployed along the power transmission channel to obtain raw data covering the power transmission channel scenario and form a power transmission dataset. Here, the multi-source sensing units may include fixed monitoring cameras and inspection drones, which can achieve full-scene, multi-view data coverage, ensure the comprehensiveness of the collected data, and avoid the omission of hidden danger information due to a single perspective.

[0061] Subsequently, this embodiment can perform preprocessing operations on the power transmission dataset based on the server. By cleaning the data, fuzzy, damaged, and redundant invalid data are removed. Then, after professional annotation, structured power transmission scene images and power transmission scene text are formed. The power transmission scene images contain visual feature information such as target location and category, while the power transmission scene text is generated by professional inspection personnel using natural language descriptions. It covers global semantic and business information such as distance from the line and safety risk level, providing a high-quality and structured data foundation for subsequent dual-modal feature extraction.

[0062] Next, the server can build a dedicated multimodal model, which includes two core components: an image feature extractor based on a convolutional neural network and a text feature extractor based on a Transformer architecture. It can be noted that the convolutional neural network has powerful local feature capture capabilities, making it suitable for extracting concrete visual features such as equipment status and target contours from power transmission scene images; the Transformer architecture excels at capturing global dependencies in sequential data, enabling deep analysis of semantic relationships and business logic in power transmission scene text. Through the collaborative design of the two extractors, accurate extraction of visual and semantic features is achieved separately, laying the model foundation for subsequent fusion analysis.

[0063] Next, the server can extract features from the power transmission scene images based on the constructed image feature extractor, obtaining image feature maps. The image feature extractor uses the backbone of the YOLOv13 target detection model. Its optimized feature pyramid and attention mechanism can efficiently capture local detail features at multiple scales, making it particularly suitable for the accurate identification of small and occluded targets in complex power transmission channel scenarios. The output image feature maps can fully present key visual information such as the target's location and shape, providing accurate basis for hazard location. Figure 2 This diagram shows a schematic of the backbone of the corresponding target detection model YOLOv13 in this embodiment;

[0064] Then, the server can further extract features from the power transmission scenario text based on the text feature extractor to obtain text feature vectors. It can be noted that the text feature extractor adopts the BERT model structure, and its bidirectional Transformer encoding can simultaneously focus on the dependency relationship between each subword and its left and right contexts, deeply integrate the global semantic information of the entire text, and accurately parse the business semantics of the power transmission scenario text such as the level of hidden danger and environmental correlation. The generated text feature vector can quantify the abstract semantic information and provide semantic support for the comprehensive judgment of hidden dangers.

[0065] Finally, the server can fuse the extracted image feature maps and text feature vectors. Through deep cross-modal information linkage, visual features and semantic features complement each other. Specifically, the image feature maps provide accurate target location information, while the text feature vectors provide global semantic judgment basis, effectively avoiding the limitations of single-modal data. Subsequently, the fused features are input into the Transformer layer of the multimodal model for feature transformation, and the final output is a detection result containing the recognition and location information of the corresponding image target and the text semantic judgment information. Here, the detection result not only clarifies the specific location of the hidden danger, but also combines business semantics to judge its risk level and impact range, significantly improving the accuracy, interpretability, and business fit of the detection result, effectively mitigating the problems of false detection and missed detection, providing reliable technical support for intelligent inspection of power transmission channels, and meeting the high requirements of accuracy and robustness of hidden danger identification.

[0066] Furthermore, in this embodiment, the aforementioned "collecting data based on multi-source sensing units and preprocessing the obtained power transmission data set of the corresponding power transmission channel to obtain power transmission scene images and power transmission scene text" may also include the following steps:

[0067] The multi-source sensing units arranged along the power transmission channel are controlled to collect data on the power transmission channel, resulting in a power transmission dataset consisting of original images and original videos.

[0068] Data cleaning is performed on the power transmission dataset to obtain a cleaned dataset.

[0069] Content annotation is performed on the cleaned dataset to obtain power transmission scene images and power transmission scene text.

[0070] For example, in this embodiment, the preprocessing process can be implemented using the following methods and steps:

[0071] First, the server can control various multi-source sensing units arranged along the power transmission channel to collect data on the channel. Specifically, the multi-source sensing units are arranged systematically along the transmission channel; for example, fixed monitoring cameras are deployed at preset intervals, and inspection drones fly along planned routes. This ensures that the data collection range covers the entire transmission channel, avoiding scene omissions due to disordered sensor unit distribution. Simultaneously, each sensing unit can collect raw images and raw videos, forming a power transmission dataset containing both types of data. Here, raw images capture static hazard details, such as equipment corrosion and hanging foreign objects, while raw videos record dynamic hazard processes, such as the movement of construction machinery and tree growth. These two types of data complement each other, comprehensively reflecting the state of the power transmission channel scene and providing rich feature sources for subsequent multimodal detection.

[0072] Next, the server can perform data cleaning based on the collected power transmission dataset to obtain a cleaned dataset. The cleaning process targets interference information in the original data: removing blurry images / video frames caused by data transmission damage or focus misalignment, removing invalid data with severe occlusion or missing content, and deduplicating data with repeated collection and redundant viewpoints to ensure that the cleaned dataset retains only clear, complete, and effective scene information, avoiding interference data from affecting the model's learning of effective features, and laying a high-quality data foundation for subsequent annotation and model training.

[0073] Finally, the server can further annotate the cleaned dataset to obtain power transmission scene images and text. Here, the annotation process employs a dual-dimensional structured annotation approach, combining visual and semantic elements. Specifically, bounding boxes are used to annotate the target locations and category information of the cleaned original images, forming power transmission scene images. This ensures that the image data contains accurate local target features. For example, target locations might be the xmin, xmax, ymin, and ymax coordinates of construction machinery, while category information might be, for example, cranes or excavators. For each power transmission scene image, professional power inspectors generate power transmission scene text using natural language descriptions. The text must cover global semantics and business information; for example, "The crane is located 30 meters below the power line, its boom is not extended, and it does not pose a safety hazard," achieving semantic alignment between the image and text. Structured annotation gives the data clear feature labels and semantic relationships, allowing subsequent multimodal models to directly extract accurate image and text features. This avoids feature extraction bias caused by unannotated or poorly labeled data, significantly improving the accuracy and robustness of subsequent hazard identification.

[0074] The following table 1 shows the content information obtained after content annotation:

[0075]

[0076] Table 1

[0077] Furthermore, in this embodiment, the aforementioned "controlling the multi-source sensing units arranged along the power transmission channel to collect data from the power transmission channel to obtain a power transmission dataset consisting of original images and original videos" may further include the following steps:

[0078] Identify the multi-source sensing units arranged along the power transmission channel and obtain the sensing range of each multi-source sensing unit.

[0079] Each multi-source sensing unit is controlled to acquire video data based on its sensing range.

[0080] The video acquisition data is decomposed into track data to obtain image track data and audio track data;

[0081] The audio track data is type-identified, and based on the identification results, it is determined that there is an audio segment in the audio track data corresponding to any human voice type, and the audio percentage of the audio segment corresponding to the audio track data is determined.

[0082] If the audio proportion is greater than a preset proportion, each captured image frame that makes up the image track data is determined as the original image; otherwise, the video captured data is determined as the original video.

[0083] The original image and original video corresponding to each multi-source sensing unit are combined to obtain the power transmission dataset.

[0084] For example, in this embodiment, the acquisition of the power transmission data set can be achieved based on the following method steps:

[0085] First, the server can identify the multi-source sensing units arranged along the power transmission channel and obtain the sensing range of each unit. Based on the aforementioned information, the multi-source sensing units include fixed monitoring cameras and inspection drones. The fixed cameras are deployed according to the spacing of the power transmission channel segments, and their sensing range covers a fixed radius area, such as 50 meters. The inspection drones fly along a preset route, and their sensing range matches the flight altitude and camera angle, such as a 100-meter aerial photography range. Clearly defining the sensing range avoids data redundancy caused by overlapping collection areas of different sensing units, while ensuring seamless connection of the sensing ranges of each unit, covering the entire power transmission channel without any blind spots, thus laying the foundation for comprehensive acquisition of channel information in the future.

[0086] Next, the server can control each multi-source sensing unit to collect data based on the sensing range, and obtain video acquisition data. Here, the video acquisition data can synchronously record the dynamic scene of the power transmission channel and the accompanying audio, containing richer information related to potential hazards.

[0087] Subsequently, the server can perform track decomposition on the video acquisition data to obtain image track data and audio track data. It can be noted that the image track data consists of continuously acquired image frames of the video and can extract static target features, such as mechanical appearance and location; the audio track data contains sound information in the video and may contain semantic clues of dynamic hazards, such as dialogue corresponding to the start of crane operation. Track decomposition realizes the separation and processing of audio and video data, providing a data foundation for subsequent classification.

[0088] Then, the server can perform type identification on the audio track data. The identification process distinguishes between audio segments of human voice type and audio segments of non-human voice type by extracting audio features. Furthermore, based on the identification results, the server determines that there are audio segments of any human voice type in the audio track data. The server can then calculate the audio proportion of the audio segment corresponding to the audio track data. Here, the audio proportion = total duration of human voice type audio segments / total duration of audio track data. In this case, human voice type audio segments may contain business semantics related to potential hazards. For example, if an inspector records an excavator operating near a pole, the audio proportion can quantify the importance of human voice information and provide a basis for subsequent data classification.

[0089] Then, if the audio proportion is greater than a preset proportion, where the preset proportion can be set based on business needs, for example, 30%, it means that human voice information has reference value for hazard judgment, indicating that the audio track data contains more key semantic information, and the dynamic correlation of the video needs to be preserved. At this time, the server can determine each captured image frame that makes up the image track data as the original image. That is, the original image can be used for static target positioning, and the semantics can be supplemented by the corresponding audio-to-text information. Conversely, if the audio proportion is less than or equal to the preset proportion, it means that the audio information has no key value, and the dynamic features of the video are more important. The video captured data is determined as the original video. That is, the original video can be used to analyze the dynamic hazard process, such as whether trees swaying in the wind touch the line.

[0090] Finally, the server can combine the original images and original videos corresponding to each multi-source sensing unit to obtain the power transmission dataset. This classification method ensures that the types of the original images and original videos match the requirements of the hidden danger features, avoids feature loss caused by data type mismatch, and provides suitable data support for subsequent models to accurately extract features and improve the accuracy and robustness of hidden danger identification.

[0091] Furthermore, in this embodiment, since the data classification process of the power transmission channel is based solely on whether the audio proportion is greater than a preset proportion to determine the data type, without considering the dynamic characteristics of human voice audio segments, such as time distribution and decibel changes, classification bias is likely to occur. That is, for example, although some audio proportions exceed the preset proportion, human voice segments are discretely distributed and decrease in decibels, and do not actually have continuous key semantics, but are still split into original images, resulting in the loss of dynamic hidden danger related information. Therefore, in order to solve this technical problem, the following steps may also be included:

[0092] In response to the audio proportion being greater than a preset proportion, the audio track data is divided into nodes at corresponding preset time intervals to obtain the trend determination points;

[0093] In response to any trend determination point being located in an audio segment corresponding to any human voice type, the trend determination point is designated as a trend identifier point, and the audio decibel value corresponding to each trend identifier point is determined based on the audio segment.

[0094] Based on the chronological order, the time interval between trend marker points in adjacent positions is determined. In response to the trend marker points showing a gradually increasing trend based on the time interval and a gradually decreasing trend based on the audio decibel value, the video acquisition data is determined as the original video. Conversely, each acquired image frame that makes up the image track data is determined as the original image.

[0095] For example, in this embodiment, to improve classification accuracy, it can be achieved based on the following method steps:

[0096] First, in response to the situation where the audio proportion is greater than the preset proportion, that is, the audio contains a certain amount of human voice information, and its value needs to be further judged, the server can divide the audio track data into nodes corresponding to the preset time interval to obtain each trend determination point. Here, the preset time interval is set based on the dynamic change frequency of the power transmission channel hidden dangers. For example, it can be 10 seconds / interval, which can avoid the redundancy of points caused by too short an interval, and ensure that the trend is not missed due to too long an interval. The trend determination points can accurately mark the state of different time points in the audio track, providing quantitative nodes for subsequent analysis of the time distribution and characteristic changes of human voice segments, avoiding the one-sidedness of judgment caused by relying only on the overall proportion.

[0097] Next, if any trend identification point is located in an audio segment corresponding to any human voice type, it indicates that the audio at that time point contains semantic information. The server can then mark the trend identification point as a trend identifier point and simultaneously determine the audio decibel value of each trend identifier point based on the audio segment. It can be noted that the audio decibel value can reflect the clarity and importance of human voices. For example, when inspectors report potential hazards, the decibel level is stable and relatively high, while irrelevant conversations have lower decibel levels and fluctuate greatly. By marking trend identifier points and collecting decibel values, audio nodes that truly contain key semantics can be filtered out, and discrete, low-value human voice segments can be excluded from interference, providing accurate data for subsequent trend analysis.

[0098] Subsequently, the server can determine the time interval between trend marker points located in adjacent positions based on the chronological order. The time interval reflects the temporal distribution density of human voice segments. The smaller the interval, the more continuous the human voice and the stronger the semantic correlation. At the same time, the server can determine the decibel change trend of each trend marker point based on the audio decibel value, which can be increasing, decreasing, or stable.

[0099] In one scenario, the response determines that the trend markers based on the time interval between points gradually increase, meaning that the time interval between human voice segments increases and the dispersion increases. Conversely, the trend markers based on the decibel value of the audio gradually decrease, meaning that the human voice becomes weaker and the semantic value decreases. This indicates that the key semantics in the audio are discrete and weakened, and there is no need to retain the dynamic correlation of the video. In this case, the server can determine the video acquisition data as the original image, meaning that the static target feature extraction can be focused on based on the original image, avoiding the impact of dynamic redundant data on model efficiency.

[0100] Conversely, in another scenario, if the time interval between points shows a gradually decreasing trend, or the decibel value shows a gradually increasing / stabilizing trend, it indicates that the audio contains continuous key semantics, and the dynamic information of the video needs to be preserved. In this case, the server can determine each acquired image frame that makes up the image track data as the original video. That is, the original video can completely record the dynamic hidden danger process, providing data support for subsequent semantic analysis of dynamic hidden dangers, realizing refined judgment of data classification, ensuring that the type classification of the original image and the original video completely matches the needs of hidden danger characteristics, avoiding classification deviation caused by a single proportion judgment, laying the foundation for accurate feature extraction by multimodal models, and further improving the accuracy and robustness of hidden danger identification in power transmission channels.

[0101] Furthermore, in this embodiment, the image feature extractor is the object detection model YOLOv13, wherein the image feature extractor includes:

[0102] The system consists of two convolutional neural network layers (Conv), one DS-C3k2 layer, and the DS-C3k2 layer is followed by one Conv layer, one DS-C3k2 layer, one depthwise separable convolutional layer (DSConv), one A2C2f layer, one DSConv layer, one A2C2f layer, and two upsample layers.

[0103] For example, in this embodiment, the image feature extractor is specifically the object detection model YOLOv13, with the following specific structure and function:

[0104] first, Figure 3 The structure diagram of DS-C3k2 in YOLOv13 is as follows: Figure 3 As shown, this includes structures corresponding to DSConv, DS-Bottleneck, DS-C3k, and DS-C3k2. Figure 4 This is the structure diagram of A2C2f in YOLOv13, as shown below. Figure 4 As shown, including the corresponding A2C2f structure and the corresponding AttentionBlock structure, it can be explained that the network starts with two convolutional neural network Conv layers. The Conv layers use 3×3 convolutional kernels and appropriate strides to perform preliminary feature extraction on the input power transmission scene image, filtering redundant noise in the image, such as tree shadows and light interference in complex backgrounds, while retaining basic visual features such as equipment outlines and line directions. Then, a DS-C3k2 layer is connected. This layer is composed of multiple DS-C3k modules and is built on a depthwise separable convolutional layer. While ensuring the accuracy of feature extraction, it significantly reduces the number of model parameters and computational complexity, adapting to the computing power limitations of power transmission inspection equipment. At the same time, the modular design of the DS-C3k2 layer can enhance the feature representation ability of medium-scale targets in the power transmission scene, laying the foundation for subsequent multi-scale feature fusion. Medium-scale targets can be, for example, towers and small and medium-sized construction machinery.

[0105] Secondly, after initial feature extraction, a single Conv layer is used to adjust the feature dimensions, ensuring that the number of channels in the feature map matches the subsequent layers. Next, a DS-C3k2 layer is connected to further enhance the extraction of local detail features in the power transmission scenario, avoiding the omission of potential hazards due to insufficient feature representation. Then, a depthwise separable convolutional layer, DSConv, is connected. This layer further compresses the model's computational load while maintaining the spatial resolution of the feature map, ensuring no loss of target location information. Afterward, two A2C2f layers are connected consecutively. The A2C2f layer integrates a region attention mechanism and cross-stage feature fusion: the region attention mechanism models the dependencies between adjacent regions in the spatial dimension, such as the association between towers and surrounding construction machinery, enhancing the feature weights of key hazard areas; cross-stage feature fusion uses residual jump branches to fuse low-level detail features with high-level semantic features.

[0106] Finally, after mid-term feature enhancement, two upsampling layers are set at the end of the network to enlarge the size of high-level feature maps through bilinear interpolation and other methods. The upsampling operation can achieve dimensional alignment of feature maps at different levels. For example, the high-level semantic feature map that has been downsampled multiple times is adjusted to the same size as the initially extracted low-level detail feature map, ensuring that the multi-scale features can accurately correspond to the target position in the power transmission scenario during subsequent feature fusion, and avoiding positioning deviations caused by feature map size mismatch.

[0107] Furthermore, in this embodiment, the depth-separable convolutional layer DSConv includes:

[0108] The system consists of a depthwise convolutional layer (DWConv), a pointwise convolutional layer (PWConv), a batch normalization layer (BatchNorm), and an activation function (SiLU).

[0109] For example, in this embodiment, the depthwise separable convolutional layer can improve the feature extraction accuracy while reducing the computational cost by splitting the convolution operation and introducing normalization and activation mechanisms. The depthwise separable convolutional layer includes a depthwise convolutional layer, a pointwise convolutional layer, a batch normalization layer and an activation function in sequence according to the feature processing flow. Each layer works together to achieve efficient and accurate extraction of image features in the power transmission scene.

[0110] First, the deep convolutional layer, as the first core module of the DSConv layer, uses a single-channel convolutional kernel to perform channel-independent convolution operations on the input power transmission scene feature map. Unlike the fully connected operation of the traditional convolutional layer with multiple inputs and multiple outputs, the DWConv layer allocates a separate convolutional kernel to each input channel and extracts features only from the spatial information within a single channel. For example, it can separate spatial details such as line outlines and construction machinery edges from the power transmission scene feature map.

[0111] Secondly, a pointwise convolutional layer is connected after the deep convolutional layer. This layer uses a 1×1 convolutional kernel to perform cross-channel convolution operations on the feature map output by the DWConv layer. Here, the core role of the PWConv layer is to fuse the single-channel spatial features extracted by the DWConv layer. It can associate and integrate the features of different channels through the weight allocation of the 1×1 convolutional kernel to generate a feature map containing multi-channel collaborative information.

[0112] Next, a batch normalization layer is connected after the pointwise convolutional layer. This layer standardizes the feature map output by the PWConv layer. That is, by calculating the mean and variance of each batch of feature data, the feature values ​​are mapped to a standard distribution interval with a mean of 0 and a variance of 1. It can be noted that in the power transmission channel scenario, different lighting and different shooting angles will lead to large differences in the distribution of image features. The BatchNorm layer can effectively eliminate this distribution shift, avoid the problem of training fluctuation or decreased generalization ability of the model due to unstable feature distribution, ensure that the features of power transmission scenarios under different environments can be stably extracted, and improve the robustness of hazard identification.

[0113] Finally, the last step of the DSConv layer sets the activation function to perform a nonlinear transformation on the feature map output by the batch normalization layer. Here, the SiLU function achieves smooth nonlinear activation by multiplying the input value by the Sigmoid function. Compared with the traditional ReLU function, it can still maintain a certain gradient output when the input value is negative, avoiding the phenomenon of neuron death. In the feature extraction of power transmission scenarios, the SiLU layer can enhance the nonlinear correlation in the feature map, such as the nonlinear relationship between the boom angle of construction machinery and the safety distance of the line, highlight the feature response of key hidden danger features, suppress the interference of irrelevant background features, and further improve the accuracy of feature extraction, providing a high-quality visual feature foundation for subsequent multimodal fusion and hidden danger identification.

[0114] Furthermore, in this embodiment, the DS-C3k2 layer is composed of multiple DS-C3k modules, and is constructed entirely based on the depth-separable convolutional layer DSConv.

[0115] For example, in this embodiment, the DS-C3k2 layer consists of multiple DS-C3k modules, using a combination of parallel and serial approaches. Each DS-C3k module has independent local feature extraction capabilities. Through the combination of multiple modules, a hierarchical extraction and collaborative fusion feature processing mechanism can be formed. For instance, the earlier DS-C3k modules focus on small-scale targets in power transmission scenarios, such as corrosion of line connectors and hanging foreign objects, capturing detailed texture features through fine convolutional kernels. The later DS-C3k modules focus on extracting the contour and spatial location features of medium to large targets. In addition, the DS-C3k2 layer as a whole is built based on depthwise separable convolutional layers, and the core convolutional operations of all DS-C3k modules rely on DSCon. The v-layer implementation, namely the depthwise separable convolutional layer, significantly reduces computational complexity and parameter count through the splitting operations of depthwise convolution and pointwise convolution, ensuring efficient feature extraction even on lightweight hardware such as drones and edge terminals. At the same time, the DSConv layer maintains high feature extraction accuracy while reducing computational costs: depthwise convolution is responsible for capturing spatial details of power transmission scene images, such as the boom angle of construction machinery and the direction of the line, while pointwise convolution is responsible for cross-channel feature fusion, such as associating tower features with surrounding environmental features to determine whether there is a risk of insufficient safety distance. The two work together to ensure that the extracted features contain both accurate local details and global channel correlation information, effectively improving the ability to express the features of potential hazards in power transmission scenes.

[0116] Furthermore, in this embodiment, the text feature extractor is composed of a text processing model BERT structure, and the text feature extractor includes: a vector embedding module, a Transformer encoder, and a text output unit;

[0117] The vector embedding module is configured to encode the power transmission scenario text after it has been segmented into sub-words by the word segmenter to obtain the input embedding vector;

[0118] The Transformer encoder includes a self-attention mechanism and a feedforward neural network. The self-attention mechanism is configured to capture dependencies between text sequences, and the feedforward neural network is configured to perform feature transformation on the output of the self-attention mechanism.

[0119] The text output unit is configured to output the probability distribution of randomly masked subwords in the input sequence, and the continuous binary classification probability distribution between two sentences.

[0120] For example, in this embodiment, Figure 5 The diagram shows the structure of the BERT bidirectional Transformer in this embodiment. A text feature extractor is constructed using the BERT structure, enabling efficient extraction of global semantic and business-related features from text in power transmission scenarios. The specific structure and functions are as follows:

[0121] First, the vector embedding module, as the starting point for text feature extraction, encodes the power transmission scenario text after it has been segmented into sub-words by the word segmenter, obtaining the input embedding vector. In the power transmission scenario, the text content contains technical terms and natural language descriptions. The word segmenter first breaks down this type of text into fine-grained sub-words, and then the embedding layer maps each sub-word into a fixed-dimensional vector, thereby transforming abstract text symbols into computable numerical vectors while preserving the basic semantic information of the sub-words.

[0122] Secondly, the Transformer encoder is the core module of the text feature extractor, containing a self-attention mechanism and a feedforward neural network. The two work together to achieve deep parsing of the global semantics of power transmission scenario text. Here, the self-attention mechanism can simultaneously pay attention to the semantic information of all other sub-words in the text sequence when processing each sub-word, capturing cross-positional dependencies. For example, when processing power transmission scenario text corresponding to a crane located 30 meters below the line, with its boom not open, and not constituting a serious hazard, the self-attention mechanism will establish semantic associations between the crane and sub-words such as "30 meters below the line," "boom not open," and "not constituting a serious hazard," accurately determining the logical relationship between the crane's position, state, and hazard level. This avoids the limitation of traditional unidirectional models that can only partially understand the text, ensuring the comprehensiveness of semantic understanding. Furthermore, after the self-attention mechanism outputs semantic association features, the feedforward neural network will perform non-linear transformation and dimensional mapping on these features to extract higher-level semantic features.

[0123] Subsequently, the text output unit is responsible for transforming the semantic features processed by the Transformer encoder into specific output results. During the model training phase, some words in the input text are randomly masked. For example, the phrase "the crane is located below the power line" is masked as "[MASK] is located below the power line." The text output unit outputs the probability distribution of the possible words corresponding to each masked position, which can verify the model's accuracy in understanding the semantics of the power transmission scenario text. Furthermore, power transmission scenario texts often contain multiple sentences with related information. For example, the first sentence is "the crane is located 30 meters below the power line," and the second sentence is "the crane's boom is not open." The text output unit outputs the binary classification probability of whether these two sentences are continuous semantics. For example, the probability of being continuous is 0.95, and the probability of being discontinuous is 0.05. This strengthens the contextual logical connection of the text, avoids the judgment bias of potential hazards caused by the semantic breaks of sentences, and ensures that the text features can fully reflect the global semantic logic of the power transmission scenario.

[0124] Furthermore, in this embodiment, the aforementioned "fusion of the image feature map and text feature vector, and output of detection results based on a multimodal model" may further include the following steps:

[0125] The image feature map and the text feature vector are concatenated and fused, and the fused content is input into the Transformer layer of the multimodal model for feature transformation to obtain the detection result.

[0126] For example, in this embodiment, the acquisition of the detection results can be achieved based on the following method steps:

[0127] First, the server can perform concatenation and fusion processing on the image feature map obtained based on the image feature extractor and the text feature vector obtained based on the text feature extractor. It can be explained that in the power transmission channel scenario, the image feature map contains the spatial location and visual detail features of the target, such as the local features of the bounding box coordinates and boom shape of the corresponding crane. The text feature vector contains global semantic and business-related features, such as the semantic information that the crane is located 30 meters below the line and the boom is not open, which does not constitute a serious hidden danger. Here, the concatenation and fusion is not a simple dimensional superposition, but through feature alignment technology, such as mapping the text feature vector to the same dimensional space as the image feature map, and associating and concatenating the two modal features according to the correspondence of spatial location and semantic description. For example, the image features of the corresponding crane bounding box are accurately bound to the text features of the corresponding crane not in operation, forming a fused content containing both visual and semantic information. This ensures that the fused content retains the precise positioning advantage of the image and has the global semantic support of the text, avoiding the one-sided judgment of hidden dangers caused by isolated features.

[0128] Next, the server can input the fused content obtained after splicing and merging into the Transformer layer of the multimodal model for feature transformation. It can be noted that the Transformer layer has a powerful global feature association capability. Its self-attention mechanism can further explore the deep logical relationship between image position features and text semantic features in the fused content. For example, when processing the fused content of crane positioning and non-operation semantics, the Transformer layer will strengthen the semantic association between the boom not being opened and no safety risk, while verifying the matching of the crane position and the safe distance of the line, and eliminating the misjudgment of dangerous position but safe state.

[0129] Finally, through feature transformation, the fused content is further refined into a structured feature representation, ultimately outputting detection results containing the identification and positioning information of the corresponding image target and the text semantic judgment information. Among them, the identification and positioning information of the image target clarifies the category and spatial coordinates of the hidden danger target, while the text semantic judgment information clarifies the business attributes of the hidden danger. For example, if the distance to the line is 30 meters and the boom is not open, the safety level is general. Thus, management personnel do not need to interpret the image and text information separately to quickly grasp the complete situation of the hidden danger, which greatly improves the accuracy and robustness of hidden danger identification. For example, when facing a crane under the line, the detection result can simultaneously clarify where the crane is and whether it constitutes a hidden danger, avoiding delays in handling due to fragmented information. The spatial coordinates are, for example, xmin, xmax, ymin, and ymax.

[0130] The following section provides further explanation using practical application scenarios, such as... Figure 6 As shown, when a crane appears under the roadway, the image feature extractor extracts the crane's information and predicts its range, i.e., the coordinates of the target box. However, the text extractor extracts information indicating that the crane is close to the roadway but not in operation, meaning its boom is not extended and does not pose a serious safety hazard; the hazard is classified as a general hazard. Therefore, this result will not send an alarm message to the specific business specialist. An alarm message will only be sent to the specific business specialist when the boom is extended and operation begins, and corresponding measures will be taken. Table 2 shows some alarm messages used for the recognition results.

[0131]

[0132] Table 2

[0133] In summary, this embodiment uses YOLOv13 as the backbone network for image feature extraction. While ensuring high-precision target detection capabilities, it also fully considers the lightweight design of the model. Specifically, YOLOv13 further optimizes the feature pyramid and attention mechanism in its structure, enabling efficient capture of local detail features at multiple scales. It is particularly suitable for accurate identification of small and occluded targets in complex scenarios such as power transmission channels. Its lightweight architecture significantly reduces computational overhead, making the image feature extraction and inference process faster, meeting the dual requirements of real-time performance and deployment efficiency in practical engineering. Furthermore, for text feature extraction, this embodiment introduces the BERT model, fully utilizing its bidirectional Transformer encoding structure to achieve a deep semantic understanding of the input text. Unlike traditional methods that rely solely on word order or local windows, BERT can simultaneously consider the dependencies between each word and its surrounding context, and integrate global semantic information of the entire text. By collaboratively utilizing image features extracted by YOLOv13 and text semantic features generated by BERT, it achieves deep fusion of cross-modal information. This framework not only overcomes the inherent defects of traditional pure visual detection models that focus only on local aspects and ignore semantics, but also effectively alleviates the practical problem of technically correct detection results that are irrelevant to business applications. In typical application scenarios such as intelligent inspection of power transmission channels, this embodiment can combine the equipment status in the image with the operation and maintenance context in the text, significantly improving the accuracy, interpretability, and business relevance of the detection results, providing reliable technical support for the intelligent operation and maintenance of power systems.

[0134] Another embodiment of the present invention provides a multimodal detection system for power transmission channel scenarios that integrates global semantics and local features. Figure 7 Its corresponding system block diagram includes:

[0135] The data acquisition module is configured to acquire data based on a multi-source sensing unit and preprocess the obtained power transmission data set of the corresponding power transmission channel to obtain power transmission scene images and power transmission scene text.

[0136] The model building module is configured to build a multimodal model, wherein the multimodal model includes an image feature extractor based on a convolutional neural network and a text feature extractor based on a Transformer architecture;

[0137] The image extraction module is configured to extract features from the power transmission scene image based on the image feature extractor to obtain an image feature map;

[0138] The text extraction module is configured to extract features from the power transmission scenario text based on a text feature extractor to obtain a text feature vector;

[0139] The feature fusion module is configured to fuse the image feature map and the text feature vector, and output the detection result based on the multimodal model. The detection result includes the recognition and positioning information of the corresponding image target and the text semantic judgment information.

[0140] In the specification provided herein, the algorithms and displays are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used with the examples of this invention. The required structure for constructing such systems is apparent from the above description. Furthermore, this invention is not directed to any particular programming language. It should be understood that the contents of the invention described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing preferred embodiments of the invention.

[0141] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0142] Similarly, it should be understood that, in order to streamline this disclosure and aid in understanding one or more of the various aspects of the invention, in the above description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof.

[0143] Those skilled in the art will understand that modules, units, or components of the devices disclosed in the examples herein can be arranged in the devices described in this embodiment, or alternatively, can be located in one or more devices different from the devices in this example. The modules in the foregoing examples can be combined into a single module or, in addition, can be divided into multiple sub-modules.

[0144] Those skilled in the art will understand that the modules in the device of the embodiment can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiment can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components.

[0145] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features included in other embodiments but not others, combinations of features from different embodiments are meant to be within the scope of the invention and form different embodiments.

[0146] Furthermore, some of the embodiments described herein are methods or combinations of method elements that can be implemented by a processor of a computer system or by other means of performing the functions. Therefore, a processor having the necessary instructions for implementing the methods or method elements forms means for implementing the methods or method elements. Furthermore, the elements described herein in the apparatus embodiments are examples of means for implementing the functions performed by elements for the purposes of carrying out the invention.

[0147] As used herein, unless otherwise specified, the use of ordinal numbers such as “first,” “second,” “third,” etc., to describe ordinary objects merely indicates different instances of similar objects and is not intended to imply that the objects being described must have a given order in time, space, ordering, or any other manner.

[0148] Although the invention has been described with respect to a limited number of embodiments, those skilled in the art will understand from the foregoing description that other embodiments are conceivable within the scope of the invention described herein. Furthermore, it should be noted that the language used in this specification has been chosen primarily for readability and edibility purposes, and not for the purpose of explaining or limiting the subject matter of the invention.

Claims

1. A multimodal detection method for power transmission channel scenarios that integrates global semantics and local features, characterized in that, include: Data is collected based on multi-source sensing units, and the power transmission datasets of the corresponding power transmission channels are preprocessed to obtain power transmission scene images and power transmission scene text. Construct a multimodal model, wherein the multimodal model includes an image feature extractor based on a convolutional neural network and a text feature extractor based on a Transformer architecture; Based on the image feature extractor, feature extraction is performed on the power transmission scene image to obtain an image feature map; Based on the text feature extractor, feature extraction is performed on the power transmission scenario text to obtain a text feature vector; The image feature map and text feature vector are fused, and the detection result is output based on a multimodal model. The detection result includes the recognition and positioning information of the corresponding image target and the text semantic judgment information.

2. The method according to claim 1, characterized in that, Data is collected using multi-source sensing units, and the resulting power transmission data sets for the corresponding transmission channels are preprocessed to obtain power transmission scene images and text, including: The multi-source sensing units arranged along the power transmission channel are controlled to collect data on the power transmission channel, resulting in a power transmission dataset consisting of original images and original videos. Data cleaning is performed on the power transmission dataset to obtain a cleaned dataset. Content annotation is performed on the cleaned dataset to obtain power transmission scene images and power transmission scene text.

3. The method according to claim 2, characterized in that, The multi-source sensing units arranged along the power transmission channel are controlled to collect data from the transmission channel, resulting in a power transmission dataset consisting of original images and original videos, including: Identify the multi-source sensing units arranged along the power transmission channel and obtain the sensing range of each multi-source sensing unit. Each multi-source sensing unit is controlled to acquire video data based on its sensing range. The video acquisition data is decomposed into track data to obtain image track data and audio track data; The audio track data is type-identified, and based on the identification results, it is determined that there is an audio segment in the audio track data corresponding to any human voice type, and the audio percentage of the audio segment corresponding to the audio track data is determined. If the audio proportion is greater than a preset proportion, each captured image frame that makes up the image track data is determined as the original image; otherwise, the video captured data is determined as the original video. The original image and original video corresponding to each multi-source sensing unit are combined to obtain the power transmission dataset.

4. The method according to claim 3, characterized in that, The method further includes: In response to the audio proportion being greater than a preset proportion, the audio track data is divided into nodes at corresponding preset time intervals to obtain the trend determination points; In response to any trend determination point being located in an audio segment corresponding to any human voice type, the trend determination point is designated as a trend identifier point, and the audio decibel value corresponding to each trend identifier point is determined based on the audio segment. Based on the chronological order, the time interval between trend marker points in adjacent positions is determined. In response to the trend marker points showing a gradually increasing trend based on the time interval and a gradually decreasing trend based on the audio decibel value, the video acquisition data is determined as the original video. Conversely, each acquired image frame that makes up the image track data is determined as the original image.

5. The method according to claim 1, characterized in that, The image feature extractor is the object detection model YOLOv13, wherein the image feature extractor includes: The system consists of two convolutional neural network layers (Conv), one DS-C3k2 layer, and the DS-C3k2 layer is followed by one Conv layer, one DS-C3k2 layer, one depthwise separable convolutional layer (DSConv), one A2C2f layer, one DSConv layer, one A2C2f layer, and two upsample layers.

6. The method according to claim 5, characterized in that, The depth-separable convolutional layer DSConv includes: The system consists of a depthwise convolutional layer (DWConv), a pointwise convolutional layer (PWConv), a batch normalization layer (BatchNorm), and an activation function (SiLU).

7. The method according to claim 5, characterized in that, The DS-C3k2 layer consists of multiple DS-C3k modules and is constructed based on the depth-separable convolutional layer DSConv.

8. The method according to claim 1, characterized in that, The text feature extractor is composed of the BERT text processing model structure, and the text feature extractor includes: a vector embedding module, a Transformer encoder, and a text output unit; The vector embedding module is configured to encode the power transmission scenario text after it has been segmented into sub-words by the word segmenter to obtain the input embedding vector; The Transformer encoder includes a self-attention mechanism and a feedforward neural network. The self-attention mechanism is configured to capture dependencies between text sequences, and the feedforward neural network is configured to perform feature transformation on the output of the self-attention mechanism. The text output unit is configured to output the probability distribution of randomly masked subwords in the input sequence, and the continuous binary classification probability distribution between two sentences.

9. The method according to claim 1, characterized in that, The image feature map and text feature vector are fused, and the detection result is output based on a multimodal model, including: The image feature map and the text feature vector are concatenated and fused, and the fused content is input into the Transformer layer of the multimodal model for feature transformation to obtain the detection result.

10. A multimodal detection system for power transmission channel scenarios that integrates global semantics and local features, characterized in that, include: The data acquisition module is configured to acquire data based on a multi-source sensing unit and preprocess the obtained power transmission data set of the corresponding power transmission channel to obtain power transmission scene images and power transmission scene text. The model building module is configured to build a multimodal model, wherein the multimodal model includes an image feature extractor based on a convolutional neural network and a text feature extractor based on a Transformer architecture; The image extraction module is configured to extract features from the power transmission scene image based on the image feature extractor to obtain an image feature map; The text extraction module is configured to extract features from the power transmission scenario text based on a text feature extractor to obtain a text feature vector; The feature fusion module is configured to fuse the image feature map and the text feature vector, and output the detection result based on the multimodal model. The detection result includes the recognition and positioning information of the corresponding image target and the text semantic judgment information.

Images