UAV control system and method for mounting robot

Abstract

The present application relates to the field of intelligent perception control technology, and specifically discloses a drone control system and method for mounting robots, which uses machine vision technology based on deep learning to analyze the image of the mounting state of the mounting robot mounted on the cable, extracts multi-level robot mounting state features, and intelligently determines whether the mounting state meets the preset requirements, so as to generate corresponding safety warning prompts. In this way, intelligent monitoring and judgment of the mounting state of the mounting robot can be achieved, effectively improving the safety and efficiency of the mounting robot's high-altitude operations.

Description

Technical Field

[0001] The present application relates to the field of intelligent perception control technology, and more specifically, to a drone control system and method for mounting a robot. Background Art

[0002] In the power system, substations are the core power transmission and distribution hubs, and their safety and stability are crucial to the operation of the entire power system. Traditional substation inspection and maintenance work usually relies on manual labor, which is not only inefficient, but also faces safety risks brought by high-altitude operations. With the continuous development of science and technology, especially the integrated application of drones and mounted robots, a new solution is provided for substation inspection, maintenance and fault detection.

[0003] In recent years, using drones to accurately place mounting robots on transmission lines for high-altitude operations has become an important direction for the development of substation automation and intelligence. This "drone + robot" operation method can not only greatly improve operation efficiency and reduce labor costs, but also effectively avoid the safety risks brought by high-altitude operations. However, in the "drone + robot" operation mode, how to ensure that the mounting robot can be accurately and stably mounted on the designated transmission line in a complex substation environment is a key issue currently faced.

[0004] The layout of the transmission lines in the substation is complex and changeable, and there are many interference factors such as electromagnetic interference and wind impact, which increase the difficulty and risk of mounting operations. Most of the existing related technical research and application practices focus on the hardware design and optimization of drones and mounting robots, lacking intelligent perception of the robot's mounting status. This leads to the difficulty of timely and effective discovery and handling of potential safety hazards in the actual operation process once unstable mounting or other abnormal conditions occur, which may lead to safety accidents. Therefore, a drone control system and method for mounting robots is expected to achieve intelligent detection and early warning of the robot's mounting status, so as to improve the accuracy and safety of the mounting robot's high-altitude operations. Summary of the invention

[0005] In order to solve the above technical problems, this application is proposed. The embodiment of this application provides a drone control system and method for a mounted robot, which uses machine vision technology based on deep learning to perform image analysis on the mounting state image of the mounted robot mounted on the cable, extracts multi-level robot mounting state features, and intelligently determines whether the mounting state meets the preset requirements, so as to generate corresponding safety warning prompts. In this way, intelligent monitoring and judgment of the mounting state of the mounted robot can be achieved, effectively improving the safety and efficiency of the high-altitude operation of the mounted robot.

[0006] Accordingly, according to one aspect of the present application, a drone control system for mounting a robot is provided, comprising:

[0007] A mounting state image acquisition module is used to acquire a mounting state image of the mounting robot mounted on the cable captured by a camera of the drone;

[0008] A mounting state multi-scale perception module, used for performing multi-scale perception on the mounting state image to obtain a mounting state shallow feature map and a mounting state semantic feature map;

[0009] A multi-scale attention enhancement module, used for inputting the mounting state semantic feature map into a feature map enhancer based on a multi-scale attention mask mechanism to obtain an enhanced mounting state semantic feature map;

[0010] A semantic-guided fusion module, used for performing semantic-guided fusion on the enhanced mount state semantic feature map and the mount state shallow feature map to obtain a semantic-guided mount state multi-channel integrated representation map;

[0011] The safety warning module is used to determine whether to generate a safety warning prompt based on the semantically guided multi-channel integrated representation diagram of the mounting status.

[0012] In the above-mentioned drone control system for mounting a robot, the mounting state multi-scale perception module is used to: pass the mounting state image through a mounting state multi-scale feature sensor based on a pyramid network to obtain the mounting state shallow feature map and the mounting state semantic feature map.

[0013] In the above-mentioned unmanned aerial vehicle control system for mounting robots, the multi-scale attention enhancement module includes: a multi-scale downsampling unit, which is used to perform multi-scale downsampling on the mounting state semantic feature map to obtain a sequence of downsampled mounting state semantic feature maps; an attention scoring unit, which is used to perform attention scoring on the sequence of downsampled mounting state semantic feature maps based on a soft attention mechanism to obtain a sequence of attention scoring coefficients; a normalization processing unit, which is used to normalize the sequence of attention scoring coefficients through a Softmax function to obtain a sequence of importance weight factors; a multi-scale feature weighted fusion unit, which is used to use the sequence of importance weight factors as a weight sequence to calculate the weighted sum of the sequence of downsampled mounting state semantic feature maps to obtain the enhanced mounting state semantic feature map.

[0014] In the above-mentioned unmanned aerial vehicle control system for mounting robots, the semantic-guided fusion module includes: a multi-scale correlation feature extraction unit, which is used to extract the multi-scale correlation features between the enhanced mounting state semantic feature map and the mounting state shallow feature map to obtain the mounting state deep semantic-shallow detail correlation feature multi-scale representation vector; a guided fusion unit, which is used to use the feature values ​​of each position in the mounting state deep semantic-shallow detail correlation feature multi-scale representation vector as weights to perform weighted fusion on the enhanced mounting state semantic feature map and the mounting state shallow feature map to obtain the semantic-guided mounting state multi-channel integrated representation map.

[0015] In the above-mentioned drone control system for mounting robots, the multi-scale correlation feature extraction unit includes: a feature cascade subunit, which is used to cascade the enhanced mounting state semantic feature map and the mounting state shallow feature map to obtain a mounting state deep semantic-shallow detail series feature map; an association encoding subunit, which is used to perform convolution encoding on the mounting state deep semantic-shallow detail series feature map to obtain a mounting state deep semantic-shallow detail correlation feature map; a data distribution balancing processing subunit, which is used to perform batch normalization processing on the mounting state deep semantic-shallow detail correlation feature map to obtain a normalized mounting state deep semantic-shallow detail correlation feature map; a multi-scale pooling subunit, which is used to perform batch normalization processing on the normalized mounting state deep semantic-shallow detail correlation feature map. The feature map is subjected to maximum-based global pooling processing, random-value-based global pooling processing and average-based global pooling processing to obtain a global maximum pooling feature vector of deep semantics-shallow details associated with the mounting state, a global mean pooling feature vector of deep semantics-shallow details associated with the mounting state and a global random value pooling feature vector of deep semantics-shallow details associated with the mounting state; a weighted fusion subunit is used to calculate the position-weighted sum of the global maximum pooling feature vector of deep semantics-shallow details associated with the mounting state, the global mean pooling feature vector of deep semantics-shallow details associated with the mounting state and the global random value pooling feature vector of deep semantics-shallow details associated with the mounting state to obtain a multi-scale representation vector of the deep semantics-shallow details associated feature of the mounting state.

[0016] In the above-mentioned drone control system for mounting robots, the guided fusion unit is used to: use the eigenvalues ​​of each position in the multi-scale representation vector of the mounting state deep semantics-shallow details associated features as weights, and weight each feature matrix along the channel dimension of the mounting state deep semantics-shallow details series feature map to obtain the semantically guided mounting state multi-channel integrated representation map.

[0017] In the above-mentioned drone control system for mounting robots, the safety warning module includes: a mounting state recognition unit, which is used to pass the semantically guided mounting state multi-channel integrated representation diagram through a classifier-based mounting state identifier to obtain a recognition result, and the recognition result is used to indicate whether the mounting state meets the preset requirements; a warning prompt unit, which is used to generate the safety warning prompt in response to the recognition result that the mounting state does not meet the preset requirements.

[0018] In the above-mentioned drone control system for mounting robots, it also includes a training module for training the pyramid network-based mounting state multi-scale feature sensor, the multi-scale attention mask mechanism-based feature map enhancer, the semantically guided multi-channel feature integration module and the classifier-based mounting state identifier.

[0019] In the above-mentioned unmanned aerial vehicle control system for mounting robots, the training module includes: a training data acquisition unit, which is used to acquire training data, wherein the training data includes a training mounting state image of the mounting robot mounted on a cable collected by a camera of the unmanned aerial vehicle, and a real value of whether the mounting state meets the preset requirements; a training data multi-scale perception unit, which is used to pass the training mounting state image through the mounting state multi-scale feature perception device based on the pyramid network to obtain a training mounting state shallow feature map and a training mounting state semantic feature map; a training data semantic reinforcement unit, which is used to input the training mounting state semantic feature map into the feature map reinforcer based on the multi-scale attention mask mechanism to obtain a training reinforced mounting state semantic feature map; a training data feature fusion unit, which is used to input the training reinforced mounting state semantic feature map into the feature map reinforcer based on the multi-scale attention mask mechanism to obtain a training reinforced mounting state semantic feature map. The semantic feature map of the mounted state and the shallow feature map of the trained mounted state are input into the semantic-guided multi-channel feature integration module to obtain the multi-channel integrated representation map of the trained semantic-guided mounted state; a classification loss unit is used to pass the multi-channel integrated representation map of the trained semantic-guided mounted state through the mount state identifier based on the classifier to obtain a classification loss function value; a model training unit is used to train the multi-scale feature perceptron of the mounted state based on the pyramid network, the feature map enhancer based on the multi-scale attention mask mechanism, the semantic-guided multi-channel feature integration module and the mount state identifier based on the classifier with the classification loss function value, wherein in each round of iteration of the training, the multi-channel integrated representation map of the trained semantic-guided mounted state is iteratively optimized.

[0020] According to another aspect of the present application, a method for controlling a drone for mounting a robot is provided, comprising:

[0021] Obtaining a mounting state image of the mounting robot mounted on the cable captured by a camera of the drone;

[0022] Performing multi-scale perception on the mounting state image to obtain a mounting state shallow feature map and a mounting state semantic feature map;

[0023] Inputting the mounting state semantic feature map into a feature map enhancer based on a multi-scale attention mask mechanism to obtain an enhanced mounting state semantic feature map;

[0024] Performing semantic-guided fusion on the enhanced mount state semantic feature map and the mount state shallow feature map to obtain a semantic-guided mount state multi-channel integrated representation map;

[0025] Based on the semantically guided multi-channel integrated representation diagram of the mounting status, it is determined whether to generate a safety warning prompt.

[0026] Compared with the prior art, the drone control system and method for mounting robots provided by the present application uses machine vision technology based on deep learning to analyze the image of the mounting state of the mounting robot mounted on the cable, extracting multi-level robot mounting state features, thereby intelligently judging whether the mounting state meets the preset requirements, so as to generate corresponding safety warning prompts. In this way, intelligent monitoring and judgment of the mounting state of the mounting robot can be achieved, effectively improving the safety and efficiency of the high-altitude operation of the mounting robot. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0028] Figure 1 4 is a block diagram of a drone control system for mounting a robot according to an embodiment of the present application.

[0029] Figure 2 Schematic diagram of the architecture of a drone control system for mounting a robot according to an embodiment of the present application.

[0030] Figure 3 It is a block diagram of a multi-scale attention enhancement module in a drone control system for mounting a robot according to an embodiment of the present application.

[0031] Figure 4 It is a block diagram of a semantic guidance fusion module in a UAV control system for mounting a robot according to an embodiment of the present application.

[0032] Figure 5 It is a block diagram of a safety warning module in a drone control system for mounting a robot according to an embodiment of the present application.

[0033] Figure 6 It is a block diagram of a training module in a drone control system for mounting a robot according to an embodiment of the present application.

[0034] Figure 7 Flow chart of a method for controlling a drone mounted with a robot according to an embodiment of the present application. DETAILED DESCRIPTION

[0035] Below, the embodiments of the present application will be described in more detail in conjunction with the accompanying drawings, and the above and other purposes, features and advantages of the present application will become more apparent. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the example embodiments described herein.

[0036] As mentioned in the above background technology, in terms of the integrated application of drones and mounted robots, most of the existing related technologies focus on the hardware design and optimization of drones and mounted robots, but lack intelligent perception of the robot's mounting status. This leads to the difficulty in timely and effective discovery and handling of potential safety hazards in the actual operation process once unstable mounting or other abnormal conditions occur, which may lead to safety accidents. In response to the above technical problems, the technical concept of this application is to use machine vision technology based on deep learning to perform image analysis on the mounting state image of the mounting robot mounted on the cable, extract multi-level robot mounting state features, and intelligently judge whether the mounting state meets the preset requirements, so as to generate corresponding safety warning prompts. In this way, intelligent monitoring and judgment of the mounting state of the mounting robot can be achieved, effectively improving the safety and efficiency of the mounting robot's aerial operations.

[0037] Figure 1 4 is a block diagram of a drone control system for mounting a robot according to an embodiment of the present application. Figure 2 FIG. 1 is a schematic diagram of the architecture of a drone control system for mounting a robot according to an embodiment of the present application. Figure 1 and Figure 2As shown, according to an embodiment of the present application, a drone control system 100 for a mounting robot includes: a mounting state image acquisition module 110, which is used to obtain a mounting state image of the mounting robot mounted on a cable collected by a camera of the drone; a mounting state multi-scale perception module 120, which is used to perform multi-scale perception on the mounting state image to obtain a mounting state shallow feature map and a mounting state semantic feature map; a multi-scale attention enhancement module 130, which is used to input the mounting state semantic feature map into a feature map enhancer based on a multi-scale attention mask mechanism to obtain an enhanced mounting state semantic feature map; a semantic guided fusion module 140, which is used to perform semantic guided fusion on the enhanced mounting state semantic feature map and the mounting state shallow feature map to obtain a semantic guided mounting state multi-channel integrated representation map; a safety warning module 150, which is used to determine whether to generate a safety warning prompt based on the semantic guided mounting state multi-channel integrated representation map.

[0038] In the above-mentioned drone control system 100 for mounting robots, the mounting state image acquisition module 110 is used to obtain the mounting state image of the mounting robot mounted on the cable captured by the camera of the drone. It should be understood that since the drone and the mounting robot are operating at high altitudes, the operator may not be able to directly observe the state of the mounting robot. Therefore, by collecting the mounting state image of the mounting robot mounted on the cable through the camera deployed on the drone, the direct information of the physical interaction between the mounting robot and the cable, such as the stability of the contact point, the bending degree of the cable, etc., can be clearly displayed, providing an important source of information for the subsequent mounting state safety warning, so as to timely detect any potential unstable or dangerous situations and ensure the safety of the mounting robot operation process.

[0039] In the above-mentioned drone control system 100 for mounting robots, the mounting state multi-scale perception module 120 is used to perform multi-scale perception on the mounting state image to obtain a shallow feature map of the mounting state and a semantic feature map of the mounting state. In a specific example of the present application, the multi-scale perception of the mounting state image is implemented by passing the mounting state image through a multi-scale feature sensor of the mounting state based on a pyramid network to obtain the shallow feature map of the mounting state and the semantic feature map of the mounting state. It should be understood that, considering that in the mounting state detection of the robot, the interaction between the mounting robot and the cable may involve feature information of different levels, such as local details of the mounting point, the overall bending degree of the cable, etc. Therefore, in order to comprehensively extract the mounting state information of the robot, in the technical solution of the present application, a multi-scale feature sensor of the mounting state based on a pyramid network is used to perform multi-scale perception on the mounting state image. Specifically, the pyramid network is a network structure that represents images at multiple scales. It constructs feature pyramids of different scales and performs multi-scale downsampling and upsampling operations on the mounting state image to achieve multi-scale and multi-level perception of the mounting state, thereby capturing the shallow feature map of the mounting state and the semantic feature map of the mounting state. The shallow feature map of the mounting state mainly reflects the local detail information of the robot's mounting state, while the semantic feature map of the mounting state focuses more on the global semantic context information of the robot's mounting state, providing a comprehensive feature representation for subsequent mounting state judgment.

[0040] In the above-mentioned drone control system 100 for mounting robots, the multi-scale attention enhancement module 130 is used to input the mounting state semantic feature map into a feature map enhancer based on a multi-scale attention mask mechanism to obtain an enhanced mounting state semantic feature map. It should be understood that since the mounting state semantic feature map may contain several information elements that are irrelevant to the mounting state discrimination, such as background noise and irrelevant objects. Therefore, in order to further improve the semantic expression ability of the mounting state semantic feature map, in the technical solution of the present application, a feature map enhancer based on a multi-scale attention mask mechanism is used to enhance the mounting state semantic feature map. Specifically, the multi-scale attention mask mechanism learns the importance and relevance of different scale features in the mounting state semantic feature map, adaptively adjusts the weights of different levels of features in the mounting state semantic feature map, and dynamically generates an attention mask to enhance the attention to objects and features of different sizes in the mounting state image, thereby achieving the enhancement and prominence of key feature information and the suppression of irrelevant interference information, so as to improve the semantic expression ability and discrimination accuracy of the mounting state semantic feature map.

[0041] Figure 3FIG. 1 is a block diagram of a multi-scale attention enhancement module in a drone control system for mounting a robot according to an embodiment of the present application. Figure 3 As shown, the multi-scale attention enhancement module 130 includes: a multi-scale downsampling unit 131, which is used to perform multi-scale downsampling on the mount state semantic feature map to obtain a sequence of downsampled mount state semantic feature maps; an attention scoring unit 132, which is used to perform attention scoring on the sequence of downsampled mount state semantic feature maps based on a soft attention mechanism to obtain a sequence of attention scoring coefficients; a normalization processing unit 133, which is used to normalize the sequence of attention scoring coefficients through a Softmax function to obtain a sequence of importance weight factors; a multi-scale feature weighted fusion unit 134, which is used to use the sequence of importance weight factors as a weight sequence to calculate the weighted sum of the sequence of downsampled mount state semantic feature maps to obtain the enhanced mount state semantic feature map.

[0042] In the above-mentioned drone control system 100 for mounting robots, the semantic guided fusion module 140 is used to perform semantic guided fusion on the enhanced mounting state semantic feature map and the mounting state shallow feature map to obtain a semantic guided mounting state multi-channel integrated representation map. It should be understood that the enhanced mounting state semantic feature map and the mounting state shallow feature map respectively describe the mounting state of the mounting robot from the deep semantic level and the shallow detail level. By effectively fusing and integrating the two, the feature representation of the mounting state can be further enriched, and the accuracy of mounting state detection can be improved. However, traditional feature fusion techniques, such as simple weighted summation at the pixel level or direct splicing between channels, may ignore the potential differences between features at the semantic level and the detail level. Therefore, in order to optimize the feature fusion effect of the enhanced mounting state semantic feature map and the mounting state shallow feature map, in the technical solution of the present application, a semantic guided multi-channel feature integration module is introduced to fuse the enhanced mounting state semantic feature map and the mounting state shallow feature map. Specifically, the semantically guided multi-channel feature integration module uses the high-level semantic association features between the enhanced mounting state semantic feature map and the mounting state shallow feature map to guide the feature fusion process, generates attention guidance through high-level semantic association information, promotes the effective combination of features at different levels, and fully considers the complementarity between semantic level and detail level features, so that the model can more accurately identify and focus on key foreground objects in the image, which not only enhances the discriminative ability of feature representation, but also improves the model's recognition accuracy for subtle structures in the image, which helps to improve the accuracy and robustness of mounting state detection of the mounting robot.

[0043] Figure 4FIG. 1 is a block diagram of a semantic guidance fusion module in a drone control system for mounting a robot according to an embodiment of the present application. Figure 4 As shown, the semantic guided fusion module 140 includes: a multi-scale correlation feature extraction unit 141, which is used to extract the multi-scale correlation features between the enhanced mounting state semantic feature map and the mounting state shallow feature map to obtain the mounting state deep semantic-shallow detail correlation feature multi-scale representation vector; a guided fusion unit 142, which is used to use the feature values ​​of each position in the mounting state deep semantic-shallow detail correlation feature multi-scale representation vector as weights to perform weighted fusion on the enhanced mounting state semantic feature map and the mounting state shallow feature map to obtain the semantic guided mounting state multi-channel integrated representation map.

[0044] Specifically, the multi-scale correlation feature extraction unit 141 includes: a feature cascade subunit, which is used to cascade the enhanced mounting state semantic feature map and the mounting state shallow feature map to obtain a mounting state deep semantic-shallow detail series feature map; an association encoding subunit, which is used to perform convolution encoding on the mounting state deep semantic-shallow detail series feature map to obtain a mounting state deep semantic-shallow detail correlation feature map; a data distribution balancing processing subunit, which is used to perform batch normalization processing on the mounting state deep semantic-shallow detail correlation feature map to obtain a normalized mounting state deep semantic-shallow detail correlation feature map; a multi-scale pooling subunit, which is used to perform the most recent multi-scale pooling processing on the normalized mounting state deep semantic-shallow detail correlation feature map. Large value global pooling processing, random value-based global pooling processing and average value-based global pooling processing are performed to obtain the global maximum pooling feature vector associated with the mounting state deep semantics and shallow details, the global mean pooling feature vector associated with the mounting state deep semantics and shallow details, and the global random value pooling feature vector associated with the mounting state deep semantics and shallow details; a weighted fusion subunit is used to calculate the position-weighted sum between the global maximum pooling feature vector associated with the mounting state deep semantics and shallow details, the global mean pooling feature vector associated with the mounting state deep semantics and shallow details, and the global random value pooling feature vector associated with the mounting state deep semantics and shallow details to obtain the multi-scale representation vector of the mounting state deep semantics and shallow details association features.

[0045] That is, the multi-scale correlation feature extraction unit 141 is used to process the enhanced mounting state semantic feature map and the mounting state shallow feature map using the following multi-scale correlation feature extraction formula to obtain the mounting state deep semantic-shallow detail correlation feature multi-scale representation vector, wherein the multi-scale correlation feature extraction formula is:

[0046]

[0047] in, represents the enhanced mounting state semantic feature graph, represents the shallow feature map of the mounting state, Indicates serial processing, represents the convolution process, represents batch normalization processing, It is the normalized mounting state deep semantics-shallow detail correlation feature map, represents the global pooling process based on the average value, Represents global pooling processing based on the maximum value, Represents global pooling processing based on random values, For different weight parameters, represents the sum of corresponding factors, It is a multi-scale representation vector of the deep semantics-shallow detail association features of the mounting state.

[0048] Specifically, the guided fusion unit 142 is used to: use the feature values ​​of each position in the multi-scale representation vector of the mounting state deep semantics-shallow details associated features as weights, and weight each feature matrix along the channel dimension of the mounting state deep semantics-shallow details series feature map to obtain the semantically guided mounting state multi-channel integrated representation map.

[0049] In the above-mentioned drone control system 100 for mounting a robot, the safety warning module 150 is used to determine whether to generate a safety warning prompt based on the semantically guided multi-channel integrated representation diagram of the mounting state. Figure 5 FIG. 1 is a block diagram of a safety warning module in a drone control system for mounting a robot according to an embodiment of the present application. Figure 5 As shown, the safety warning module 150 includes: a mounting state recognition unit 151, which is used to pass the semantic-guided mounting state multi-channel integrated representation diagram through a classifier-based mounting state identifier to obtain a recognition result, and the recognition result is used to indicate whether the mounting state meets the preset requirements; a warning prompt unit 152, which is used to generate the safety warning prompt in response to the recognition result that the mounting state does not meet the preset requirements.

[0050] Specifically, the mounting state recognition unit 151 is used to pass the semantically guided mounting state multi-channel integrated representation diagram through a classifier-based mounting state recognizer to obtain a recognition result, and the recognition result is used to indicate whether the mounting state meets the preset requirements. It should be understood that the semantically guided mounting state multi-channel integrated representation diagram obtained after the above-mentioned feature integration processing contains the semantic information and detailed features of the robot mounting state, which is an important basis for judging the mounting state. To this end, in the technical solution of the present application, a classification algorithm is further used to perform feature recognition and classification on the semantically guided mounting state multi-channel integrated representation diagram to determine whether the mounting robot is correctly mounted on the cable according to the preset requirements. In a specific implementation, the classifier-based mounting state recognizer learns the classification boundaries and feature laws of the mounting state by training a large amount of sample data, so as to achieve accurate recognition of the mounting state.

[0051] In the technical solution of the present application, when the semantically guided integration of the multi-scale and multi-depth image semantic features of the mounting status image is performed on the semantically guided multi-channel integrated representation diagram of the semantically guided mounting status, due to the semantic guidance differences of the image semantic features in different dimensions, the local distribution differences of the semantically guided multi-channel integrated representation diagram of the semantically guided mounting status cause the predetermined category harmony deviation, which affects the classification iteration effect.

[0052] In a preferred embodiment, the semantically guided multi-channel integrated representation diagram of the mount state is passed through a classifier-based mount state identifier to obtain an identification result, including the following steps:

[0053] Expanding the semantically guided mounting state multi-channel integrated representation graph into a semantically guided mounting state multi-channel integrated representation vector;

[0054] The semantically guided mount state multi-channel integrated representation vector is subjected to feature local distribution correction to obtain an optimized semantically guided mount state multi-channel integrated representation vector, wherein the feature local distribution correction step is specifically expressed as follows:

[0055]

[0056] in, Representation semantics guides the multi-channel integrated representation vector of the mount status, represents the transpose of a vector, represents the autocorrelation matrix, The autocorrelation matrix is ​​represented by and row vector, represents the weight matrix, The weight matrix The matrix value of the position, represents the height of the autocorrelation matrix, represents the set of real numbers, represents matrix multiplication, represents dot product, Representation optimized semantically guided mount state multi-channel integrated representation vector, represents the length of the multi-channel integrated representation vector of the semantically guided mount status, express function;

[0057] The optimized semantically guided mounting state multi-channel integrated representation vector is passed through the classifier-based mounting state identifier to obtain the identification result.

[0058] Based on the self-correlation dimension of the semantically guided mount state multi-channel integrated representation vector as the object to be modulated, association enhancement is performed based on the complexity of the spatial secondary dimensions of the feature distribution pattern in the multi-dimensional feature space, thereby incorporating the displacement of the decomposable association dimension of the semantically guided mount state multi-channel integrated representation vector into the heterogeneous embedding space, thereby enhancing its association self-consistency through the collaborative fine-tuning of the decomposable dimension set represented by the heterogeneous embedding space, so as to improve the predetermined category harmony of the feature set of the semantically guided mount state multi-channel integrated representation vector in the associated target classification domain, thereby improving the classification training speed and the accuracy of the classification results when the semantically guided mount state multi-channel integrated representation graph is classified by the mount state identifier.

[0059] Specifically, the early warning prompt unit 152 is used to generate the safety early warning prompt in response to the recognition result that the mounting state does not meet the preset requirements. That is, when the mounting state identifier determines that the mounting state does not meet the preset requirements, the safety early warning prompt mechanism will be automatically triggered to attract the attention of the operator. In the embodiment of the present application, the safety early warning prompt includes but is not limited to sound alarms, flashing lights, screen pop-ups, etc., to ensure that the operator can promptly discover the instability or potential danger of the mounting state, and take corresponding measures in time to avoid accidents such as equipment damage and personal injury caused by abnormal mounting state, thereby improving the safety and reliability of the mounting robot operation.

[0060] It should be understood that before using the above neural network model, it needs to be trained. That is, in the drone control system for mounting robots of the present application, it also includes a training module for training the mounting state multi-scale feature sensor based on the pyramid network, the feature map enhancer based on the multi-scale attention mask mechanism, the semantically guided multi-channel feature integration module and the mounting state identifier based on the classifier.

[0061] Figure 6FIG. 1 is a block diagram of a training module in a drone control system for mounting a robot according to an embodiment of the present application. Figure 6 As shown, the training module 200 includes: a training data acquisition unit 210, which is used to acquire training data, wherein the training data includes a training mounting state image of a mounting robot mounted on a cable collected by a camera of an unmanned aerial vehicle, and a real value of whether the mounting state meets the preset requirements; a training data multi-scale perception unit 220, which is used to pass the training mounting state image through the mounting state multi-scale feature perception device based on the pyramid network to obtain a shallow feature map of the training mounting state and a semantic feature map of the training mounting state; a training data semantic enhancement unit 230, which is used to input the training mounting state semantic feature map into the feature map enhancer based on the multi-scale attention mask mechanism to obtain a training enhanced mounting state semantic feature map; a training data feature fusion unit 240, which is used to input the training enhanced mounting state semantic feature map into the feature map enhancer based on the multi-scale attention mask mechanism to obtain a training enhanced mounting state semantic feature map. The state semantic feature map and the shallow feature map of the training mount state are input into the semantic-guided multi-channel feature integration module to obtain the training semantic-guided mount state multi-channel integrated representation map; the classification loss unit 250 is used to pass the training semantic-guided mount state multi-channel integrated representation map through the classifier-based mount state identifier to obtain a classification loss function value; the model training unit 260 is used to train the pyramid network-based mount state multi-scale feature perceptron, the feature map enhancer based on the multi-scale attention mask mechanism, the semantic-guided multi-channel feature integration module and the classifier-based mount state identifier with the classification loss function value, wherein in each round of iteration of the training, the training semantic-guided mount state multi-channel integrated representation map is iteratively optimized.

[0062] In a preferred embodiment of the present application, when the model is trained with the classification loss function value, the training semantics-guided mounting state multi-channel integrated representation graph is iteratively optimized in each iteration of the training.

[0063] Specifically, the iterative optimization process includes the following steps: expanding the training semantic-guided mount state multi-channel integrated representation graph into a training semantic-guided mount state multi-channel integrated representation vector; determining the first eigenvalue and the second eigenvalue of any two different positions of the training semantic-guided mount state multi-channel integrated representation vector; determining the compliance probability value corresponding to the mount state meeting the preset requirements and the non-compliance probability value corresponding to the mount state not meeting the preset requirements obtained by the training semantic-guided mount state multi-channel integrated representation vector through a classifier; dividing the compliance probability value by the non-compliance probability value to obtain a semantic-guided mount state multi-channel integrated relative class probability distribution value; multiplying the compliance probability value by the non-compliance probability value to obtain a semantic-guided mount state multi-channel integrated collaborative class probability distribution value; subtracting the semantic-guided mount state multi-channel integrated relative class probability distribution value from the first eigenvalue and the second eigenvalue respectively, and then multiplying them to obtain a semantic-guided mount state multi-channel integrated collaborative class probability distribution value. Integrate relative joint representation; add the first eigenvalue and the second eigenvalue and then divide it by the semantic-guided mount state multi-channel integrated collaborative class probability distribution value to obtain the semantic-guided mount state multi-channel integrated system response representation; add the semantic-guided mount state multi-channel integrated relative joint representation and the semantic-guided mount state multi-channel integrated system response representation, and use them as the matrix value at the coordinates corresponding to different positions of the first eigenvalue and the second eigenvalue to obtain the semantic-guided mount state multi-channel integrated correction matrix; calculate the inner product of each row vector of the semantic-guided mount state multi-channel integrated correction matrix and the training semantic-guided mount state multi-channel integrated representation vector to obtain the optimized trained semantic-guided mount state multi-channel integrated representation vector; perform reverse feature aggregation on the optimized trained semantic-guided mount state multi-channel integrated representation vector according to the feature expansion method to obtain the optimized trained semantic-guided mount state multi-channel integrated representation graph.

[0064] That is, the above preferred embodiment takes the feature value of the training semantically guided multi-channel integrated representation diagram of the mount status as a unit, and performs a relativity-based joint representation and a synergy-based response representation on the feature value pairs of the training semantically guided multi-channel integrated representation diagram of the mount status as a feature set, and obtains the relative class probability distribution form and the synergy-based probability distribution form of the conforming class probability and the non-conforming class probability through the classifier. This avoids the class induction bias caused by the corresponding local distribution differences between the feature values ​​of the training semantically guided multi-channel integrated representation diagram of the mount status, so as to establish a robust understanding paradigm for class cognition of the training semantically guided multi-channel integrated representation diagram as a feature set as a whole, thereby improving the iterative effect of the training semantically guided multi-channel integrated representation diagram of the mount status as a feature set in the classification process, that is, improving the classification training speed and the accuracy of the classification results when the training semantically guided multi-channel integrated representation diagram of the mount status is classified by the mount status identifier.

[0065] In summary, the drone control system for the mounting robot according to the embodiment of the present application is explained, which uses machine vision technology based on deep learning to perform image analysis on the mounting state image of the mounting robot mounted on the cable, extracts multi-level robot mounting state features, and intelligently determines whether the mounting state meets the preset requirements, so as to generate corresponding safety warning prompts. In this way, intelligent monitoring and judgment of the mounting state of the mounting robot can be achieved, effectively improving the safety and efficiency of the mounting robot's high-altitude operations.

[0066] Figure 7 FIG. 1 is a flow chart of a method for controlling a drone mounted with a robot according to an embodiment of the present application. Figure 7 As shown, according to the embodiment of the present application, the drone control method for a mounting robot includes the following steps: S1, obtaining a mounting state image of the mounting robot mounted on a cable captured by a camera of the drone; S2, performing multi-scale perception on the mounting state image to obtain a mounting state shallow feature map and a mounting state semantic feature map; S3, inputting the mounting state semantic feature map into a feature map enhancer based on a multi-scale attention mask mechanism to obtain an enhanced mounting state semantic feature map; S4, performing semantic-guided fusion on the enhanced mounting state semantic feature map and the mounting state shallow feature map to obtain a semantic-guided mounting state multi-channel integrated representation map; S5, determining whether to generate a safety warning prompt based on the semantic-guided mounting state multi-channel integrated representation map.

[0067] Here, those skilled in the art will appreciate that the specific operations of each step in the above-mentioned method for controlling a drone with a mounted robot have been described in detail above. Figures 1 to 6The description of the drone control system for mounting the robot has been described in detail, and therefore, its repeated description will be omitted.

[0068] The basic principle of the present invention is described above in conjunction with specific embodiments. However, it should be pointed out that the advantages, strengths, effects, etc. mentioned in the present invention are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. must be possessed by each embodiment of the present invention. In addition, the specific details of the above embodiments are only for the purpose of illustration and facilitation of understanding, rather than limitation, and the above details do not limit the present invention to being implemented by adopting the above specific details.

[0069] In the above embodiments, the description of each embodiment has its own emphasis. For the parts that are not described or recorded in detail in a certain embodiment, please refer to the relevant description of other embodiments. In the several embodiments provided by the present invention, it should be understood that the disclosed system and method can be implemented in other ways. For example, the system embodiment described above is only schematic. For example, the unit division is only a logical function division, and there may be other division methods in actual implementation. The unit described as a separate component may or may not be physically separated, and the component displayed as a unit may or may not be a physical sub-unit, that is, it may be located in one place, or it may be distributed on multiple network sub-units. Some or all of the units can be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0070] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference to a figure in a claim should not be considered as limiting the claim to which it relates.

[0071] In addition, it is obvious that the word "comprising" does not exclude other sub-units or steps, and the singular does not exclude the plural. Multiple sub-units stated in the system claim can also be implemented by one sub-unit through software or hardware.

[0072] Finally, it should be noted that the above description has been given for the purpose of illustration and description. In addition, the above embodiments are only used to illustrate the technical solution of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A UAV control system for mounting a robot, characterized in that: include: A mounting state image acquisition module is used to acquire a mounting state image of the mounting robot mounted on the cable captured by a camera of the drone; A mounting state multi-scale perception module, used for performing multi-scale perception on the mounting state image to obtain a mounting state shallow feature map and a mounting state semantic feature map; A multi-scale attention enhancement module, used for inputting the mounting state semantic feature map into a feature map enhancer based on a multi-scale attention mask mechanism to obtain an enhanced mounting state semantic feature map; A semantic-guided fusion module, used for performing semantic-guided fusion on the enhanced mount state semantic feature map and the mount state shallow feature map to obtain a semantic-guided mount state multi-channel integrated representation map; A safety warning module, used to determine whether to generate a safety warning prompt based on the semantically guided multi-channel integrated representation diagram of the mounting status; Wherein, the safety warning module includes: A mount state recognition unit, used for passing the semantic guided mount state multi-channel integrated representation diagram through a classifier-based mount state recognizer to obtain a recognition result, wherein the recognition result is used to indicate whether the mount state meets the preset requirements; An early warning prompt unit, configured to generate the safety early warning prompt in response to the identification result that the mounting state does not meet the preset requirements; The step of passing the semantically guided mounting state multi-channel integrated representation graph through a classifier-based mounting state identifier to obtain an identification result comprises the following steps: Expanding the semantically guided mounting state multi-channel integrated representation graph into a semantically guided mounting state multi-channel integrated representation vector; The semantically guided mount state multi-channel integrated representation vector is subjected to feature local distribution correction to obtain an optimized semantically guided mount state multi-channel integrated representation vector, wherein the feature local distribution correction step is specifically expressed as follows: Among them, V represents the semantic guided mount state multi-channel integrated representation vector, T represents the transpose of the vector, M represents the self-correlation matrix, V i and V j represents the i-th and j-th row vectors of the autocorrelation matrix, M' represents the weight matrix, m i，j represents the matrix value of the (i, j)th position of the weight matrix, L represents the height of the autocorrelation matrix, R represents a set of real numbers, represents matrix multiplication, ⊙ represents dot multiplication, V′ represents the optimized semantic-guided mount state multi-channel integrated representation vector, length represents the length of the semantic-guided mount state multi-channel integrated representation vector, and Sigmoid represents the Sigmoid function; The optimized semantically guided mounting state multi-channel integrated representation vector is passed through the classifier-based mounting state identifier to obtain the identification result.

2. The unmanned aerial vehicle control system for mounting a robot according to claim 1, characterized in that: The mounting status multi-scale perception module is used to: The mounting state image is passed through a mounting state multi-scale feature perceptron based on a pyramid network to obtain the mounting state shallow feature map and the mounting state semantic feature map.

3. The unmanned aerial vehicle control system for mounting a robot according to claim 2, characterized in that: The multi-scale attention enhancement module includes: A multi-scale downsampling unit, used for performing multi-scale downsampling on the mounting state semantic feature map to obtain a sequence of downsampled mounting state semantic feature maps; An attention scoring unit, used for performing attention scoring on the sequence of the downsampled mounting state semantic feature graphs based on a soft attention mechanism to obtain a sequence of attention scoring coefficients; A normalization processing unit, used for normalizing the sequence of attention scoring coefficients through a Softmax function to obtain a sequence of importance weight factors; A multi-scale feature weighted fusion unit is used to calculate the weighted sum of the sequence of downsampled mounting state semantic feature maps using the sequence of importance weight factors as a weight sequence to obtain the enhanced mounting state semantic feature map.

4. The unmanned aerial vehicle control system for mounting a robot according to claim 3, characterized in that: The semantic guidance fusion module includes: A multi-scale correlation feature extraction unit, used to extract the multi-scale correlation features between the enhanced mounting state semantic feature map and the mounting state shallow feature map to obtain a mounting state deep semantic-shallow detail correlation feature multi-scale representation vector; A guided fusion unit is used to perform weighted fusion on the enhanced mounting state semantic feature map and the mounting state shallow feature map by using the feature values ​​of each position in the multi-scale representation vector of the mounting state deep semantic-shallow detail association feature as weights to obtain the semantic guided mounting state multi-channel integrated representation map.

5. The unmanned aerial vehicle control system for mounting a robot according to claim 4, characterized in that: The multi-scale correlation feature extraction unit comprises: A feature cascade subunit, used for cascading the enhanced mounting state semantic feature map and the mounting state shallow feature map to obtain a mounting state deep semantic-shallow detail tandem feature map; An associated coding subunit, used for performing convolution coding on the mounting state deep semantics-shallow detail series feature map to obtain a mounting state deep semantics-shallow detail associated feature map; A data distribution balance processing subunit is used to perform batch normalization processing on the mounting state deep semantics-shallow detail association feature map to obtain a normalized mounting state deep semantics-shallow detail association feature map; A multi-scale pooling subunit, used for performing maximum-based global pooling processing, random-value-based global pooling processing and average-based global pooling processing on the normalized mounting state deep semantics-shallow detail association feature map to obtain a mounting state deep semantics-shallow detail association global maximum pooling feature vector, a mounting state deep semantics-shallow detail association global average pooling feature vector and a mounting state deep semantics-shallow detail association global random value pooling feature vector; The weighted fusion subunit is used to calculate the position-weighted sum of the global maximum pooling feature vector of the mounting state deep semantics-shallow details association, the global mean pooling feature vector of the mounting state deep semantics-shallow details association, and the global random value pooling feature vector of the mounting state deep semantics-shallow details association to obtain the multi-scale representation vector of the mounting state deep semantics-shallow details association feature.

6. The unmanned aerial vehicle control system for mounting a robot according to claim 5, characterized in that: The guiding fusion unit is used for: Taking the eigenvalues ​​of each position in the multi-scale representation vector of the deep semantics-shallow details associated features of the mounting status as weights, each feature matrix along the channel dimension of the deep semantics-shallow details tandem feature map of the mounting status is weighted to obtain the semantically guided multi-channel integrated representation map of the mounting status.

7. The unmanned aerial vehicle control system for mounting a robot according to claim 6, characterized in that: It also includes a training module for training the pyramid network-based mounting state multi-scale feature sensor, the multi-scale attention mask mechanism-based feature map enhancer, the semantic-guided multi-channel feature integration module, and the classifier-based mounting state identifier.

8. The unmanned aerial vehicle control system for mounting a robot according to claim 7, characterized in that: The training module comprises: A training data acquisition unit, used to acquire training data, wherein the training data includes a training mounting state image of the mounting robot mounted on the cable captured by a camera of the drone, and a real value of whether the mounting state meets preset requirements; A training data multi-scale perception unit, used for passing the training mount state image through the pyramid network-based mount state multi-scale feature perception device to obtain a training mount state shallow feature map and a training mount state semantic feature map; A training data semantic enhancement unit, used for inputting the training mount state semantic feature map into the feature map enhancer based on the multi-scale attention mask mechanism to obtain a training enhanced mount state semantic feature map; A training data feature fusion unit, used for inputting the training enhanced mount state semantic feature map and the training mount state shallow feature map into the semantic-guided multi-channel feature integration module to obtain a training semantic-guided mount state multi-channel integrated representation map; A classification loss unit, used for passing the training semantics-guided mount state multi-channel integrated representation through the classifier-based mount state identifier to obtain a classification loss function value; The model training unit is used to train the pyramid network-based multi-scale feature perceptron for mounting status, the feature map enhancer based on the multi-scale attention mask mechanism, the semantic-guided multi-channel feature integration module and the classifier-based mounting status identifier with the classification loss function value, wherein in each round of iteration of the training, the training semantic-guided mounting status multi-channel integrated representation map is iteratively optimized.

9. A method for controlling a drone mounted with a robot, characterized in that: include: Obtaining a mounting state image of the mounting robot mounted on the cable captured by a camera of the drone; Performing multi-scale perception on the mounting state image to obtain a mounting state shallow feature map and a mounting state semantic feature map; Inputting the mounting state semantic feature map into a feature map enhancer based on a multi-scale attention mask mechanism to obtain an enhanced mounting state semantic feature map; Performing semantic-guided fusion on the enhanced mount state semantic feature map and the mount state shallow feature map to obtain a semantic-guided mount state multi-channel integrated representation map; Determining whether to generate a safety warning prompt based on the semantically guided multi-channel integrated representation diagram of the mounting status; Wherein, determining whether to generate a safety warning prompt based on the semantically guided multi-channel integrated representation diagram of the mounting status includes: The semantically guided multi-channel integrated representation diagram of the mounting state is passed through a classifier-based mounting state identifier to obtain a recognition result, wherein the recognition result is used to indicate whether the mounting state meets the preset requirements; In response to the identification result that the mounting state does not meet the preset requirements, generating the safety warning prompt; The step of passing the semantically guided mounting state multi-channel integrated representation graph through a classifier-based mounting state identifier to obtain an identification result comprises the following steps: Expanding the semantically guided mounting state multi-channel integrated representation graph into a semantically guided mounting state multi-channel integrated representation vector; The semantically guided mount state multi-channel integrated representation vector is subjected to feature local distribution correction to obtain an optimized semantically guided mount state multi-channel integrated representation vector, wherein the feature local distribution correction step is specifically expressed as follows: Among them, V represents the semantic guided mount state multi-channel integrated representation vector, T represents the transpose of the vector, M represents the self-correlation matrix, V i and V j represents the i-th and j-th row vectors of the autocorrelation matrix, M′ represents the weight matrix, m i，j represents the matrix value of the (i, j)th position of the weight matrix, L represents the height of the autocorrelation matrix, R represents a set of real numbers, represents matrix multiplication, ⊙ represents dot multiplication, V′ represents the optimized semantic-guided mount state multi-channel integrated representation vector, length represents the length of the semantic-guided mount state multi-channel integrated representation vector, and Sigmoid represents the Sigmoid function; The optimized semantically guided mounting state multi-channel integrated representation vector is passed through the classifier-based mounting state identifier to obtain the identification result.

Images