Unmanned aerial vehicle real-time flight path dynamic adjustment system, method and device based on artificial intelligence and medium

Abstract

The invention provides an unmanned aerial vehicle real-time flight path dynamic adjustment system and method based on artificial intelligence, equipment and a medium. The system comprises a data acquisition module for acquiring target data; the image processing module performs image preprocessing and enhancement processing on the target image data; the image segmentation module can extract image edge features in the enhanced image according to a multi-scale multi-direction morphological gradient algorithm, segment the enhanced image according to the image edge features, and merge segmented images according to the same marks; a scene recognition and classification module performs scene recognition on the processed segmented image according to a convolutional neural network model to obtain a scene type; and the path dynamic adjustment module dynamically adjusts the flight path. The scene type can be accurately judged by using an image segmentation technology, and the flight control parameters corresponding to the scene are matched, so that the accuracy of path dynamic adjustment is improved.

Description

Technical Field

[0001] This application relates to the technical field of unmanned aerial vehicle (UAV) autonomous control, and particularly to a real-time flight path dynamic adjustment system, method, electronic device, and storage medium for UAVs based on artificial intelligence. Background Art

[0002] With the rapid development of the UAV industry, the application of UAVs has become more and more in-depth in all walks of life and plays an important role. In particular, it has been widely used in military and civilian fields such as cargo sorting, equipment inspection, public security patrol, target detection, scene shooting, and power inspection. However, in various applications in both military and civilian fields, autonomous obstacle avoidance and target tracking of UAVs are often basic requirements. Therefore, it is of great practical significance to study a reliable and effective solution.

[0003] Currently, for the problem of automatic obstacle avoidance control of UAV flight paths, mainly traditional trajectory planning methods are used for obstacle avoidance. For example, the Rapidly-exploring Random Tree (RRT) and the artificial potential field algorithm, etc. However, traditional methods have their disadvantages. The RRT algorithm has a slow convergence speed and a tortuous trajectory, and the artificial potential field method has problems such as being easily trapped in local minima and oscillations. In addition, discontinuous direction commands are directly output through image information. For example, some methods are based on the hierarchical structure of the Deep Q-Network (DQN), and these hierarchical Q networks are used as high-level control strategies for navigation in different stages, including control commands such as forward, backward, left, right, and descent. This method has a low accuracy. Summary of the Invention

[0004] In view of this, it is necessary to provide a real-time flight path dynamic adjustment system, method, electronic device, and storage medium for UAVs based on artificial intelligence to accurately identify the type of application scenario and obtain a precise flight path adjustment strategy according to the type of application scenario.

[0005] In a first aspect, an embodiment of the present application provides a real-time flight path dynamic adjustment system for UAVs based on artificial intelligence. The system includes:

[0006] A data acquisition module, configured to collect target data through a UAV according to a preset instruction, where the target data includes target image data and scene data;

[0007] An image processing module, configured to perform image preprocessing and enhancement processing on the target image data;

[0008] An image segmentation module, which is used to extract image edge features in the enhanced image according to the multi-scale multi-direction morphological gradient algorithm, segment the enhanced image according to the image edge features, mark the objects in the image, merge the segmented images according to the same mark, and obtain the processed segmented image;

[0009] A scene recognition and classification module, which is used to recognize the scene of the processed segmented image according to the convolutional neural network model to obtain the scene type;

[0010] A path dynamic adjustment module, which is used to obtain a path control strategy according to the scene type and the scene data according to the scene control model, and dynamically adjust the flight path.

[0011] In one embodiment, the image processing module is used to perform image preprocessing and enhancement processing on the target image data, including:

[0012] Perform multi-dimensional orthogonal linear transformation on the target image data to obtain a reduced-dimensional image;

[0013] Successively perform downsampling and upsampling on the reduced-dimensional image to filter out noise, and use the template filtering method to perform noise reduction processing on the image to remove the noise in the image;

[0014] Perform binarization processing on the image using the adaptive threshold method to obtain a binarized image;

[0015] Convert the color image into an RGB color image according to the RGB pixel values corresponding to each pixel position;

[0016] Overlay the binarized image and the RGB color image to obtain the enhanced image.

[0017] In one embodiment, the image segmentation module is used to extract image edge features in the enhanced image according to the multi-scale multi-direction morphological gradient algorithm, segment the enhanced image according to the image edge features, mark the objects in the image, merge the segmented images according to the same mark, and obtain the processed segmented image, including:

[0018] Detect the image at multiple scales, extract the scale difference according to the scale difference formula, and the scale difference formula is:

[0019]

[0020] where 0 ≤ i ≤ n; B i represents a set of structural feature elements of size (2i - 1) × (2i + 1), and n represents the number of scales, Dilation is denoted by, erosion is denoted by Θ, and AC1 is the sample image;

[0021] The image is detected in multiple directions, and the difference in direction is extracted according to the direction difference formula. The direction difference formula is:

[0022]

[0023] where 0 ≤ j ≤ m, and m represents the number of directions;

[0024] According to the scale difference F scale (f) and the direction difference F direction (f) are weighted, the weighted sum is calculated, and a multi-scale multi-direction morphological gradient image is obtained;

[0025] Image edge features are extracted according to the multi-scale multi-direction morphological gradient image;

[0026] Image segmentation is performed according to the image edge features, and the objects in the image are labeled;

[0027] The segmented images with the same label are merged to obtain the processed segmented image.

[0028] In one embodiment, the scene recognition and classification module is used to perform scene recognition on the processed segmented image according to the convolutional neural network model to obtain the scene type, including:

[0029] Feature extraction is performed on the processed segmented image according to the feature extraction module of the convolutional neural network model to obtain the pixel features of the image;

[0030] Feature comparison is performed on the pixel features according to the scene classification module of the convolutional neural network model to obtain some scene classification types;

[0031] Weighted evaluation is performed on the some scene classification types according to the evaluation module of the convolutional neural network model to obtain the evaluated scene type.

[0032] In one embodiment, the convolutional neural network model includes a feature extraction module, a scene classification module, and an evaluation module;

[0033] Among them, the feature extraction module consists of an input layer, an output layer, 3N convolutional layers, 2N pooling layers, 2N fully connected layers, and a Gaussian connection layer. Among them, a convolutional layer is adjacent to a pooling layer, and then adjacent to a convolutional layer. Each convolutional layer is followed by a non-linear layer, and the first two non-linear layers are followed by a max pooling layer. The activation function is the Sgn function,

[0034] Among them,

[0035] The expression of the Sgn function is as follows: x is the input of the neuron;

[0036] Its loss function is L b :

[0037]

[0038] y k represents the true classification label of the k-th input sample in the input data, K represents the number of samples, and p j represents the predicted probability value of the category of the k-th input sample in the input data;

[0039] The scene classification module is composed of a fully connected layer, a non-linear layer, and a parallel processing layer combined in sequence. The parallel processing layer consists of two parts. The first part consists of a fully connected layer, a softmax layer, and a classification loss function layer, and the second part consists of a fully connected layer and a regression cost function layer;

[0040] The evaluation module performs weighted evaluation according to the output result of the scene classification module to obtain the evaluated scene type.

[0041] In one embodiment, the path dynamic adjustment module is used to obtain a path control strategy according to the scene type and the scene data based on a scene control model, and dynamically adjust the flight path, including:

[0042] Construct a scene control model according to the mapping relationship between the scene type in the sample data and the path control strategy;

[0043] Obtain the scene type, input it into the scene control model for matching, and obtain the corresponding path control strategy;

[0044] Compare the scene data with the corresponding parameters in the path control strategy to obtain the dynamic adjustment parameters of the flight data;

[0045] Regulate the flight path according to the dynamic adjustment parameters.

[0046] In one embodiment, the convolutional neural network model is obtained through training, including:

[0047] Construct an initial convolutional neural network model;

[0048] Randomly collect sample data, preprocess it and then cluster it to obtain sample data converging in different types;

[0049] Use the clustered sample data to train the initial convolutional neural network model respectively;

[0050] During the training process, when training each layer of hidden nodes, the output of the previous layer of hidden nodes is used as the input, and the output of the current layer of hidden nodes is used as the input for the next layer for pre-training;

[0051] After all the pre-training is completed, the entire network is then fine-tuned;

[0052] Through feedback regulation, an optimized convolutional neural network model is obtained.

[0053] In a second aspect, an embodiment of the present application provides a method for dynamically adjusting the real-time flight path of a drone based on artificial intelligence, which is applied to the system for dynamically adjusting the real-time flight path of a drone based on artificial intelligence as described in the first aspect. The method includes:

[0054] The drone collects target data according to a preset instruction, and the target data includes target image data and scene data;

[0055] Perform image preprocessing and enhancement processing on the target image data;

[0056] Extract the image edge features in the enhanced image according to the multi-scale multi-directional morphological gradient algorithm, segment the enhanced image according to the image edge features, mark the objects in the image, and merge the segmented images with the same mark to obtain the processed segmented image;

[0057] Perform scene recognition on the processed segmented image according to the convolutional neural network model to obtain the scene type;

[0058] According to the scene type and the scene data, obtain a path control strategy according to the scene control model and dynamically adjust the flight path.

[0059] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0060] A processor;

[0061] A memory for storing instructions executable by the processor;

[0062] Wherein, when the processor is configured to execute the instructions, it implements the method for dynamically adjusting the real-time flight path of a drone based on artificial intelligence as described in the second aspect.

[0063] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, including instructions, and the instructions direct the device to execute the method for dynamically adjusting the real-time flight path of a drone based on artificial intelligence as described in the second aspect.

[0064] A system, method, electronic device, and storage medium for dynamically adjusting the real-time flight path of an unmanned aerial vehicle based on artificial intelligence provided by an embodiment of the present application can accurately plan the flight path. The multi-scale multi-directional morphological gradient algorithm is used to extract the image edge features in the enhanced image, and the enhanced image is segmented according to the image edge features, and the objects in the image are marked. The segmented images are merged according to the same mark to obtain the processed segmented image; the processed segmented image is recognized by a convolutional neural network model to obtain the scene type; according to the scene type and the scene data, a path control strategy is obtained according to the scene control model to dynamically adjust the flight path. This solution can accurately judge the scene type and match the flight control parameters corresponding to the scene, improving the accuracy of path dynamic adjustment. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 Schematic diagram of an electronic device provided by an embodiment of the present application.

[0066] Figure 2 Schematic diagram of the framework of a system for dynamically adjusting the real-time flight path of an unmanned aerial vehicle based on artificial intelligence provided by an embodiment of the present application.

[0067] Figure 3 Schematic diagram of the flow of a method for dynamically adjusting the real-time flight path of an unmanned aerial vehicle based on artificial intelligence provided by an embodiment of the present application.

[0068] In the figure: 1 is an electronic device, 10 is a processor, 11 is a memory, 12 is a program, and 13 is a communication interface.

[0069] The realization, functional features, and advantages of the object of the present invention will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0070] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application.

[0071] It should be noted that "at least one" in the embodiments of the present application refers to one or more, and multiple refers to two or more. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which this application belongs. The terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.

[0072] It should be noted that, in the embodiments of the present application, words such as "first" and "second" are only used for the purpose of distinguishing descriptions, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying order. Features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way.

[0073] Based on the implementations in this application, all other implementations obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0074] At present, traditional detection methods mostly rely on manual inspections and regular inspections. This method is not only labor-intensive, but also limited by the frequency and efficiency of manual inspections, making it difficult to detect minor defects in a timely manner. In addition, the disadvantage of traditional methods is the lack of continuous monitoring and trend prediction of equipment operating status, making it difficult to cope with increasingly complex management needs.

[0075] In view of this, the present application provides a real-time UAV flight path dynamic adjustment system based on artificial intelligence, which can accurately judge the scene type and match the flight control parameters corresponding to the scene, thereby improving the accuracy of dynamic path adjustment.

[0076] Figure 1 This is an electronic device 10 provided by an embodiment of the present application. Figure 1 As shown, the electronic device 10 includes at least the following parts: a processor 10 and a memory 11 .

[0077] In the embodiment of the present application, the memory 11 is used to store instructions executable by the processor 10, and the processor 10 is configured to implement the following when executing the instructions: Figures 2-3 The artificial intelligence-based real-time dynamic adjustment method of the UAV flight path is shown.

[0078] Figure 2 FIG. 1 is a schematic diagram of a module of a UAV real-time flight path dynamic adjustment system based on artificial intelligence provided by an embodiment of the present application. Figure 2 The artificial intelligence-based UAV real-time flight path dynamic adjustment system 20 shown includes: a background server, a UAV, and at least the following parts: a data acquisition module 21, an image processing module 22, an image segmentation module 23, a scene recognition and classification module 24, and a path dynamic adjustment module 25.

[0079] Specifically, the control platform can be set up in a back-end server or other service systems, used to communicate with the drone, and control the flight path of the drone through instructions.

[0080] In the embodiment of the present application, the data acquisition module 21 is used to collect image data through the drone and a fixed camera, and the image data includes visible light images and infrared images.

[0081] Specifically, the data acquisition module 21 collects image data through the drone and a fixed camera, and the image data includes visible light images and infrared images. By jointly using infrared images and visible light images, more accurate and comprehensive data support can be provided, providing a richer information source for subsequent scene recognition.

[0082] In the embodiment of the present application, the image processing module 22 is used to perform image preprocessing and enhancement processing on the target image data.

[0083] Specifically, the image processing module 22 is used to perform image preprocessing and enhancement processing on the target image data, including: performing multi-dimensional orthogonal linear transformation on the target image data to obtain a reduced-dimension image; sequentially performing downsampling and upsampling on the reduced-dimension image to filter out noise, and using the template filtering method to perform noise reduction processing on the image to remove the noise in the image; performing binarization processing on the image using the adaptive threshold method to obtain a binarized image; converting the color image into an image of RGB color according to the RGB pixel values corresponding to each pixel position; superimposing the binarized image and the RGB color image to obtain an enhanced image.

[0084] After the above preprocessing and enhancement, the low-frequency information of the image is basically removed, and an image with enhanced high-frequency part is obtained, thus realizing image enhancement. At the same time, when there is an uneven illumination field, by performing binarization processing on the image, the influence of the uneven illumination field is eliminated by superimposing the sharpened image and the enhanced image. In this way, the obtained enhanced fingerprint image is relatively ideal, facilitating feature extraction during subsequent image processing, thereby reducing the probability of incorrect feature extraction and improving the accuracy of scene recognition.

[0085] The data processing module 22 uses median filtering and Gaussian filtering to remove random noise in the image. Median filtering sorts the neighborhood of each pixel value and selects the median value as the new value of the pixel, thus effectively removing salt-and-pepper noise. Gaussian filtering uses the smoothing effect of the Gaussian function to remove high-frequency noise and avoid introducing blur in the image. After denoising, the data processing module will also fuse the visible light image and the infrared image. Image fusion technology can generate more complete image information by combining the advantages of two different types of images to improve the accuracy in subsequent analysis.

[0086] Optionally, the template filtering method can also be used, or median filtering and Gaussian filtering can be used to remove random noise in the image. Median filtering sorts the neighborhood of each pixel value and selects the median value as the new value of the pixel, thus effectively removing salt-and-pepper noise. Gaussian filtering, on the other hand, uses the smoothing effect of the Gaussian function to remove high-frequency noise and avoid introducing blurring in the image. After denoising, the data processing module will also fuse the visible light image and the infrared image. Image fusion technology can generate more complete image information by combining the advantages of two different types of images, so as to improve the accuracy in subsequent analysis.

[0087] It can be understood that image denoising is to improve the accuracy of subsequent feature extraction and model training. Through the dual effects of downsampling, upsampling filtering, and using the template filtering method, the system can remove various noise interferences in the image and ensure that the obtained image is clearer. At the same time, image fusion makes the advantages of the visible light image and the infrared depth image complementary to each other, enabling the system to take into account features in multiple dimensions such as shape and spatial scale in defect recognition, thus improving the accuracy of subsequent scene recognition.

[0088] In the embodiment of the present application, the image segmentation module 23 is used to extract the image edge features in the enhanced image according to the multi-scale multi-direction morphological gradient algorithm, segment the enhanced image according to the image edge features, mark the objects in the image, and merge the segmented images according to the same mark to obtain the processed segmented image.

[0089] It can be understood that image segmentation is a necessary part of modern autonomous driving systems because an accurate understanding of the scene around the drone is crucial for navigation and motion planning. Image segmentation can help the autonomous driving of the drone identify the drivable area in a picture. Since the emergence of the Fully Convolutional Networks (FCN), convolutional neural networks have gradually become the mainstream method for processing image tasks, and many of them are directly borrowed from convolutional neural network methods in other fields. In the past decade, many scholars have made a lot of efforts in the creation of image datasets and the improvement of algorithms. Thanks to the development of deep learning theory, quite a lot of progress has been made in the subfield of visual scene understanding. The disadvantage of deep learning is that it requires a large amount of labeled data, which is time-consuming, but the advantages outweigh the disadvantages.

[0090] Specifically, in the field of drone autonomous driving, the categories of image segmentation specifically include sky, mountains, rivers, forests, high towers, people, buildings, tunnels, poles, signal towers, or others.

[0091] Specifically, the image segmentation module is used to extract the image edge features in the enhanced image according to the multi-scale multi-direction morphological gradient algorithm, segment the enhanced image according to the image edge features, mark the objects in the image, and merge the segmented images with the same mark to obtain the processed segmented image, including:

[0092] Detect the image at multiple scales, extract the scale difference according to the scale difference formula, and the scale difference formula is:

[0093]

[0094] where 0 ≤ i ≤ n; B i represents a set of structural feature elements of size (2i - 1) × (2i + 1), and n represents the number of scales, represents the dilation operation, Θ represents the erosion operation, and AC1 is the sample image;

[0095] Detect the image in multiple directions, extract the direction difference according to the direction difference formula, and the direction difference formula is:

[0096]

[0097] where 0 ≤ j ≤ m, and m represents the number of directions;

[0098] Weight the scale difference F scale (f) and the direction difference F direction (f), calculate the weighted sum to obtain the multi-scale multi-direction morphological gradient image;

[0099] Extract the image edge features according to the multi-scale multi-direction morphological gradient image;

[0100] Segment the image according to the image edge features and mark the objects in the image;

[0101] Merge the segmented images with the same mark to obtain the processed segmented image.

[0102] The above-mentioned method extracts the RGB color features of the image through the image processing module 22, combines them with the image edge features for image segmentation, and uses statistical methods to distinguish the image regions to which the extracted features belong, which can effectively improve the accuracy and speed of image segmentation. Among them, the mathematical morphology methods include dilation, erosion, opening operation and closing operation.

[0103] It can be understood that the multi-scale multi-directional morphological gradient algorithm is used to extract the image edge features in the enhanced image, and the enhanced image is segmented according to the image edge features, and the objects in the image are marked, and the segmented images are merged according to the same mark to obtain the processed segmented image; the processed segmented image is subjected to scene recognition according to the convolutional neural network model to obtain the scene type; according to the scene type and the scene data, the path control strategy is obtained according to the scene control model, and the flight path is dynamically adjusted. This solution can accurately judge the scene type, match the flight control parameters corresponding to the scene, and improve the accuracy of the path dynamic adjustment.

[0104] In the embodiment of the present application, the scene recognition and classification module 24 is used to perform scene recognition on the processed segmented image according to the convolutional neural network model to obtain the scene type.

[0105] When applying this classification model to scenarios such as unmanned aerial vehicles and autonomous driving, its application scenario determines that when an action occurs in the outside world, it is necessary to be able to react in a timely manner according to the action that occurs in the outside world. Therefore, for different scenarios, different scene recognition and classification functions are set. Only in this way, after the scene is accurately classified, can the flight action control strategy corresponding to the scene be obtained, and an accurate judgment can be made on the dynamic adjustment of the flight path.

[0106] It can be understood that the scene recognition and classification module is used to perform scene recognition on the processed segmented image according to the convolutional neural network model to obtain the scene type, including:

[0107] Feature extraction is performed on the processed segmented image according to the feature extraction module of the convolutional neural network model to obtain the pixel features of the image;

[0108] Feature comparison is performed on the pixel features according to the scene classification module of the convolutional neural network model to obtain partial scene classification types;

[0109] Weighted evaluation is performed on the partial scene classification types according to the evaluation module of the convolutional neural network model to obtain the evaluated scene type.

[0110] It can be understood that the convolutional neural network model includes a feature extraction module, a scene classification module, and an evaluation module;

[0111] Among them, the feature extraction module is composed of an input layer, an output layer, 3N convolutional layers, 2N pooling layers, 2N fully connected layers, and a Gaussian connection layer. Among them, a convolutional layer is adjacent to a pooling layer, and then adjacent to a convolutional layer. Each convolutional layer is followed by a non-linear layer, and the first two non-linear layers are followed by a max pooling layer. The activation function is the Sgn function.

[0112] Among them,

[0113] The expression of the Sgn function is: x is the input of the neuron;

[0114] Its loss function is L b :

[0115]

[0116] y k represents the true classification label of the k-th input sample in the input data, K represents the number of samples, and p j represents the class prediction probability value of the k-th input sample in the input data;

[0117] The scene classification module is composed of a fully connected layer, a non-linear layer and a parallel processing layer combined in sequence. The parallel processing layer consists of two parts. The first part consists of a fully connected layer, a softmax layer and a classification loss function layer, and the second part consists of a fully connected layer and a regression cost function layer;

[0118] The evaluation module performs weighted evaluation according to the output result of the scene classification module to obtain the evaluated scene type.

[0119] Specifically, the convolutional neural network model performs type analysis on the obtained image feature representation to distinguish different types of visual information. Analyze the image feature representation and extract the scene data. The scene data includes information such as the overall layout of the environment, the position and shape of objects, etc., which is usually achieved through image segmentation or object detection.

[0120] Specifically, the scene recognition and classification module 24 uses a convolutional neural network (CNN) to extract features from the fused image to extract texture features, color features, shape features, etc. of the defect. Through multiple layers of convolution and pooling operations, the CNN can automatically learn and extract local features in the image, and transform these features into high-level expressions through the fully connected layer for subsequent analysis and classification. In addition, the data analysis module will also use a long short-term memory network (LSTM) to construct the temporal features of the fused image, so that not only the spatial information of the image can be extracted, but also the time series features in the image can be analyzed to capture the change trend in the time dimension.

[0121] It can be understood that the convolutional neural network model is obtained through training, including: constructing an initial convolutional neural network model; randomly collecting sample data, preprocessing it and then clustering to obtain sample data converging in different types; respectively using the clustered sample data to train the initial convolutional neural network model; during the training process, when training each layer of hidden nodes, taking the output of the previous layer of hidden nodes as the input and the output of the current layer of hidden nodes as the input of the next layer for pre-training; after all pre-training is completed, fine-tuning the entire network; and obtaining an optimized convolutional neural network model through feedback adjustment.

[0122] It can be understood that a convolutional neural network can efficiently extract key information in an image. Optionally, based on the feature extraction method of an LSTM convolutional neural network, the system can comprehensively analyze in both the spatial and temporal dimensions to further improve the accuracy of classification and recognition.

[0123] In the embodiment of the present application, the path dynamic adjustment module 25 is used to obtain a path control strategy according to the scene type and the scene data based on the scene control model, and dynamically adjust the flight path.

[0124] Specifically, the path dynamic adjustment module is used to obtain a path control strategy according to the scene type and the scene data based on the scene control model, and dynamically adjust the flight path, including: constructing a scene control model according to the mapping relationship between the scene type in the sample data and the path control strategy; obtaining the scene type, inputting it into the scene control model for matching to obtain the corresponding path control strategy; comparing the scene data with the corresponding parameters in the path control strategy to obtain the dynamic adjustment parameters of the flight data; and regulating the flight path according to the dynamic adjustment parameters.

[0125] Specifically, the mapping relationship between the scene and the flight path control strategy can be designed in advance to construct a scene control model, and then the corresponding flight data can be matched according to the scene control model. Subsequently, the existing flight data can be compared with the matched corresponding flight data to determine the adjustment direction, scale, etc., and adjust in real time according to the change of the scene to improve the accuracy of autonomous control.

[0126] Figure 3 It is a schematic flow chart of the method for dynamically adjusting the real-time flight path of an unmanned aerial vehicle based on artificial intelligence provided by an embodiment of the present application. As Figure 3The method for dynamically adjusting the real-time flight path of an unmanned aerial vehicle (UAV) based on artificial intelligence as shown includes at least the following steps: S100: Collect image data through the UAV and a fixed camera; S200: Remove random noise in the image according to median filtering and Gaussian filtering, and fuse the visible light image and the infrared image to obtain a fused image; S300: Extract defect features of the fused image according to a convolutional neural network, and construct temporal features of the fused image according to a long short-term memory network; S400: Identify the defect features and temporal features according to a gradient boosting decision tree algorithm model to output the defect type and risk score; S500: Generate a defect report according to the defect type and risk score, and give an alarm when the risk score is higher than the alarm threshold.

[0127] S100: Collect image data through the UAV and a fixed camera.

[0128] In an embodiment of the present application, the method for dynamically adjusting the real-time flight path of a UAV based on artificial intelligence includes, in step S100, collecting image data through the UAV and a fixed camera, where the image data includes a visible light image and an infrared image. For the specific collection method, please refer to Figure 1 、 Figure 2 and its corresponding description, which will not be elaborated herein.

[0129] S200: Remove random noise in the image according to median filtering and Gaussian filtering, and fuse the visible light image and the infrared image to obtain a fused image.

[0130] In an embodiment of the present application, the method for dynamically adjusting the real-time flight path of a UAV based on artificial intelligence includes, in step S200, removing random noise in the image according to median filtering and Gaussian filtering, and fusing the visible light image and the infrared image to obtain a fused image. For the specific fusion method, please refer to Figure 1 、 Figure 2 and its corresponding description, which will not be elaborated herein.

[0131] S300: Extract defect features of the fused image according to a convolutional neural network to obtain defect features of the fused image, and construct temporal features of the fused image according to a long short-term memory network.

[0132] In an embodiment of the present application, the method for dynamically adjusting the real-time flight path of a UAV based on artificial intelligence includes, in step S300, extracting defect features of the fused image according to a convolutional neural network to obtain defect features of the fused image, and constructing temporal features of the fused image according to a long short-term memory network. The defect features include texture features, color features, and shape features. For the specific obtaining method, please refer to Figure 1 、 Figure 2 and its corresponding description, which will not be elaborated herein.

[0133] S400: Identify defect features and timing features according to the gradient boosting decision tree algorithm model to output the defect type and risk score.

[0134] In the embodiment of the present application, the method for dynamically adjusting the real-time flight path of the drone based on artificial intelligence includes, in step S400, identifying defect features and timing features according to the gradient boosting decision tree algorithm model to output the defect type and risk score. For the specific output method, please refer to Figure 1 、 Figure 2 and its corresponding description, which will not be elaborated herein in the present application.

[0135] S500: Generate a defect report according to the defect type and risk score, and give an alarm when the risk score is higher than the alarm threshold.

[0136] In the embodiment of the present application, the method for dynamically adjusting the real-time flight path of the drone based on artificial intelligence includes, in step S500, generating a defect report according to the defect type and risk score, and giving an alarm when the risk score is higher than the alarm threshold. For the specific generation method, please refer to Figure 1 、 Figure 2 and its corresponding description, which will not be elaborated herein in the present application.

[0137] As Figure 1 shown, the electronic device 10 at least includes the following parts: a processor 10 and a memory 11. In the embodiment of the present application, the memory 11 is used to store executable instructions of the processor 10, and the processor 10 is configured to implement the method for dynamically adjusting the real-time flight path of the drone based on artificial intelligence as Figure 2 shown when executing the instructions.

[0138] In the embodiment of the present application, a computer-readable storage medium includes instructions for instructing a device to execute the method of the first aspect. For example, the instructions instruct the device to execute the system for dynamically adjusting the real-time flight path of the drone based on artificial intelligence shown in steps S100 to S500 in Figure 2 .

[0139] The program operating in the electronic device according to an embodiment of the present application may be a program for controlling a Central Processing Unit (CPU) or the like to implement the functions of the above-described embodiments related to a solution of the present invention (a program that causes a computer to function). Then, the information processed by these devices is temporarily stored in a Random Access Memory (RAM) during its processing, and thereafter, it is stored in various ROMs such as a Read Only Memory (Flash ROM), a Hard Disk Drive (HDD), etc., and is read, corrected, and written by the CPU as needed.

[0140] It should be noted that a part of the electronic device of the above-described embodiment can also be implemented by a computer. In this case, the program for implementing the control function can be recorded on a computer-readable recording medium, and is implemented by reading the program recorded on this recording medium into the computer and executing it.

[0141] It should be noted that the "computer" mentioned here refers to a computer built into an electronic device, which is a computer including hardware such as an OS and peripheral devices. In addition, the "computer-readable recording medium" refers to a removable medium such as a floppy disk, a magneto-optical disk, a ROM, a CD-ROM, etc., and a storage device such as a hard disk built into a computer.

[0142] Moreover, the "computer-readable recording medium" may include: a medium that dynamically stores a program for a short period of time, such as a communication line in the case of transmitting a program via a network such as the Internet or a communication line such as a telephone line; a medium that stores a program for a fixed period of time, such as a volatile memory inside a computer serving as a server or a client in this case. In addition, the above program may be a program for implementing a part of the above functions, and may also be a program that can implement the above functions by being combined with a program already recorded in a computer.

[0143] In addition, the electronic device in the above-described embodiment can also be implemented as an aggregate (device group) composed of multiple devices. Each device constituting the device group may have some or all of the functions or function blocks of the electronic device of the above-described embodiment. As the device group, it is sufficient to have all the functions or function blocks of the electronic device.

[0144] It can be understood that the AI-based real-time flight path dynamic adjustment system 10 / 10a, method, electronic device, and storage medium provided by the embodiments of the present application can effectively improve the operation and maintenance level and solve the problem that traditional manual inspections cannot identify facility defects in real time and comprehensively. Through the application of this system, potential fault hazards of facilities can be quickly discovered, and corresponding repair suggestions and optimization plans can be provided through intelligent analysis, thereby minimizing equipment downtime and maintenance costs. At the same time, the adaptive ability of the system enables it to continuously learn and optimize, improve performance based on more real-time collected data, and further enhance its adaptability to different types and the ability to predict future defects.

[0145] Specifically, the specific implementation method of the above instructions by the processor 10 can refer to Figures 2 to 3 the description of the relevant steps in the corresponding embodiments and will not be elaborated here.

[0146] It should be noted that the serial numbers of the above embodiments of the present invention are only for description and do not represent the superiority or inferiority of the embodiments. And the term "including", "comprising" or any other variant thereof in this article is intended to cover a non-exclusive inclusion, so that a process, device, article or method including a series of elements not only includes those elements but also includes other elements not explicitly listed, or further includes elements inherent to such a process, device, article or method. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of another identical element in the process, device, article or method including that element.

[0147] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above and includes several instructions to enable a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in various embodiments of the present invention.

[0148] The above are only the preferred embodiments of the present invention and do not limit the patent scope of the present invention accordingly. Any equivalent structural or equivalent process transformation made by using the specification and drawings of the present invention, or directly or indirectly applied in other related technical fields, shall be equally included in the patent protection scope of the present invention.

Claims

1. A real-time flight path dynamic adjustment system for unmanned aerial vehicles based on artificial intelligence, comprising: Backend server, drone, characterized in that the system includes: A data acquisition module, used to acquire target data through a drone according to preset instructions, wherein the target data includes target image data and scene data; An image processing module is used to perform image preprocessing and enhancement processing on target image data; An image segmentation module is used to extract image edge features in the enhanced image according to a multi-scale multi-directional morphological gradient algorithm, segment the enhanced image according to the image edge features, mark objects in the image, and merge segmented images according to the same marks to obtain a processed segmented image; A scene recognition and classification module, used to perform scene recognition on the processed segmented image according to a convolutional neural network model to obtain a scene type; The path dynamic adjustment module is used to obtain a path control strategy according to the scene type and the scene data and the scene control model, and dynamically adjust the flight path.

2. The artificial intelligence-based real-time flight path dynamic adjustment system for unmanned aerial vehicles according to claim 1 is characterized in that: The image processing module is used to perform image preprocessing and enhancement processing on the target image data, including: Perform multi-dimensional orthogonal linear transformation on the target image data to obtain a reduced-dimensional image; The image after dimensionality reduction is downsampled and upsampled in turn to filter out noise, and the image is subjected to denoising by using a template filtering method to remove noise in the image; Adopting adaptive threshold method to binarize the image to obtain a binary image; Convert the color image into an RGB color image according to the RGB pixel values ​​corresponding to each pixel position; The binary image and the RGB color image are superimposed to obtain an enhanced image.

3. The artificial intelligence-based real-time flight path dynamic adjustment system for unmanned aerial vehicles according to claim 2 is characterized in that: The image segmentation module is used to extract image edge features in the enhanced image according to the multi-scale multi-directional morphological gradient algorithm, segment the enhanced image according to the image edge features, mark objects in the image, merge segmented images according to the same mark, and obtain a processed segmented image, including: Perform multiple scale detection on the image and extract the scale difference according to the scale difference formula. The scale difference formula is: Among them, 0≤i≤n; B i represents a set of structural feature elements of size (2i-1)×(2i+1), n ​​represents the number of scales, represents the dilation operation, Θ represents the erosion operation, and AC1 is the sample image; Perform multi-directional detection on the image and extract the directional difference according to the directional difference formula. The directional difference formula is: Among them, 0≤j≤m, m represents the number of directions; According to the difference in scale F scale (f) and the difference in direction F direction (f) performing weighting and calculating the weighted sum to obtain a multi-scale and multi-directional morphological gradient image; Extract image edge features based on multi-scale and multi-directional morphological gradient images; Perform image segmentation based on image edge features and mark objects in the image; The segmented images with the same label are merged to obtain the processed segmented image.

4. The artificial intelligence-based real-time flight path dynamic adjustment system for unmanned aerial vehicles according to claim 3 is characterized in that: The scene recognition and classification module is used to perform scene recognition on the processed segmented image according to the convolutional neural network model to obtain the scene type, including: Performing feature extraction on the processed segmented image according to a feature extraction module of a convolutional neural network model to obtain pixel features of the image; Performing feature comparison on the pixel features according to the scene classification module of the convolutional neural network model to obtain partial scene classification types; A weighted evaluation is performed on the partial scene classification types according to the evaluation module of the convolutional neural network model to obtain an evaluated scene type.

5. The artificial intelligence-based real-time flight path dynamic adjustment system for unmanned aerial vehicles according to claim 4 is characterized in that: The convolutional neural network model includes a feature extraction module, a scene classification module, and an evaluation module; The feature extraction module includes an input layer, an output layer, 3N convolutional layers, 2N pooling layers, 2N fully connected layers and a Gaussian connection layer, wherein a convolutional layer is connected to a pooling layer, and then to a convolutional layer. Each convolutional layer is followed by a nonlinear layer, and the first two nonlinear layers are followed by a maximum pooling layer. The activation function is the Sg n function. in, The Sg n function expression is: x is the input of the neuron; Its loss function is L b : y k represents the true classification label of the kth input sample in the input data, K represents the number of samples, and p j Represents the category prediction probability value of the kth input sample in the input data; The scene classification module is composed of a fully connected layer, a nonlinear layer and a parallel processing layer in sequence, wherein the parallel processing layer consists of two parts, the first part of which consists of a fully connected layer, a flexible maximum layer and a classification loss function layer, and the second part of which consists of a fully connected layer and a regression cost function layer; The evaluation module performs weighted evaluation according to the output results of the scene classification module to obtain the evaluated scene type.

6. The artificial intelligence-based real-time UAV flight path dynamic adjustment system according to claim 5 is characterized in that: The path dynamic adjustment module is used to obtain a path control strategy according to the scene type and the scene data and the scene control model, and dynamically adjust the flight path, including: According to the mapping relationship between the scene type and the path control strategy in the sample data, a scene control model is constructed; Obtain the scene type, input it into the scene control model for matching, and obtain the corresponding path control strategy; Comparing the scene data with the corresponding parameters in the path control strategy to obtain dynamic adjustment parameters of the flight data; The flight path is regulated based on dynamic adjustment parameters.

7. The artificial intelligence-based real-time UAV flight path dynamic adjustment system according to claim 5 is characterized in that: The convolutional neural network model is obtained through training, including: Build an initial convolutional neural network model; Randomly collect sample data, perform preprocessing and clustering, and obtain sample data that converges on different types; Using the clustered sample data to train the initial convolutional neural network model respectively; During the training process, each time a layer of hidden nodes is trained, the output of the previous layer of hidden nodes is used as input, and the output of the current layer of hidden nodes is used as input for the next layer for pre-training; After all pre-training is completed, the entire network is fine-tuned; Through feedback adjustment, the optimized convolutional neural network model is obtained.

8. A method for dynamically adjusting the real-time flight path of a UAV based on artificial intelligence, applied to the real-time dynamic flight path adjustment system of a UAV based on artificial intelligence as claimed in any one of claims 1 to 7, characterized in that: The method comprises: Collect target data by using a drone according to preset instructions, wherein the target data includes target image data and scene data; Perform image preprocessing and enhancement processing on target image data; Extracting image edge features from the enhanced image according to a multi-scale multi-directional morphological gradient algorithm, segmenting the enhanced image according to the image edge features, marking objects in the image, and merging segmented images according to the same marks to obtain a processed segmented image; Performing scene recognition on the processed segmented image according to a convolutional neural network model to obtain a scene type; According to the scene type and the scene data, a path control strategy is obtained according to a scene control model, and the flight path is dynamically adjusted.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to implement the method for dynamic adjustment of the real-time flight path of a UAV based on artificial intelligence as described in claim 8 when executing the instruction.

10. A computer-readable storage medium, characterized in that: Including instructions, which instruct the device to execute the method for dynamic adjustment of the real-time flight path of a drone based on artificial intelligence as described in claim 8.

Images