UAV flight status monitoring method and system based on deep learning

Abstract

The present application provides a method and system for monitoring the flight status of a UAV based on deep learning, which realizes a comprehensive and in-depth analysis of the flight behavior of the UAV by comprehensively using the structured and unstructured data in the UAV flight monitoring data sequence. Specifically, it can accurately identify the feature correlation between the UAV flight monitoring data sequence and the flight interaction unstructured data, thereby effectively improving the accuracy and real-time performance of flight status monitoring. Furthermore, based on the combined flight behavior analysis data, it is possible to quickly lock the target flight feedback knowledge network that best matches the target flight behavior from the huge reference flight feedback knowledge network sequence, significantly enhancing the pertinence and effectiveness of the flight feedback. Finally, the flight request task feedback report generated by the target flight feedback knowledge network not only provides important guarantees for the safe flight of the UAV, but also provides a scientific basis for the optimization of subsequent flight strategies.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and more specifically, to a method and system for monitoring the flight status of an unmanned aerial vehicle based on deep learning. Background Art

[0002] With the rapid development of drone technology, its application in aerial photography, agriculture, environmental monitoring, logistics and transportation is becoming more and more extensive. However, although the autonomous flight capability of drones is constantly improving, its flight status monitoring and feedback mechanism is still the key link to ensure flight safety and improve flight efficiency. Traditional drone flight status monitoring methods mainly rely on sensor data collected by the flight control system, such as structured information such as speed, altitude, and attitude. Although these data can intuitively reflect the flight status of the drone, it is often difficult to fully and accurately capture the actual flight behavior of the drone in the face of complex and changing flight environments.

[0003] In recent years, with the rise of big data and artificial intelligence technology, especially the rapid development of deep learning technology, new ideas have been brought to the monitoring of drone flight status. With its powerful feature extraction and pattern recognition capabilities, deep learning models have achieved remarkable results in image processing, natural language processing and other fields. However, in the field of drone flight status monitoring, how to effectively integrate the structured data and unstructured data (such as video, image, voice, etc.) generated during the drone flight process to achieve a comprehensive and in-depth analysis of the drone flight behavior is still a problem that needs to be solved.

[0004] In the existing technology, some studies have tried to apply deep learning to drone flight status monitoring, but most of them focus on processing a single type of data, ignoring the inherent connection between structured data and unstructured data. In addition, when generating flight feedback reports, there is often a lack of a systematic knowledge network as support, resulting in single feedback content and lack of pertinence. Summary of the invention

[0005] In view of the above-mentioned problems, in combination with the first aspect of the present application, an embodiment of the present application provides a method for monitoring the flight status of a drone based on deep learning, the method comprising:

[0006] Obtaining a UAV flight monitoring data sequence of at least one flight status monitoring cycle, wherein the UAV flight monitoring data sequence at least includes: storing flight interaction structured data of a flight request task of a most recent flight status monitoring cycle;

[0007] When the UAV flight monitoring data sequence also includes flight interaction unstructured data, performing a combined flight behavior analysis on the UAV flight monitoring data sequence and the flight interaction unstructured data to generate corresponding combined flight behavior analysis data, wherein the combined flight behavior analysis data is used to represent: a feature correlation between the UAV flight monitoring data sequence and the flight interaction unstructured data;

[0008] Based on the combined flight behavior analysis data, determining an associated target flight feedback knowledge network from a reference flight feedback knowledge network sequence, each reference flight feedback knowledge network in the reference flight feedback knowledge network sequence corresponding to a flight feedback knowledge point;

[0009] The target flight feedback knowledge network is used to generate a flight feedback report of the flight request task based on the UAV flight monitoring data sequence and the flight feedback knowledge points corresponding to the target flight feedback knowledge network.

[0010] In a possible implementation of the first aspect, after obtaining a UAV flight monitoring data sequence of at least one flight status monitoring cycle, the method further includes:

[0011] Based on the time series correlation information in the UAV flight monitoring data sequence, the flight interaction structured data is state reconstructed to generate flight interaction structured data after state reconstructing;

[0012] Performing flight behavior analysis on the UAV flight monitoring data sequence and the flight interaction structured data after state reconstruction to generate corresponding flight behavior analysis data;

[0013] Then, based on the combined flight behavior analysis data, determining an associated target flight feedback knowledge network from a reference flight feedback knowledge network sequence includes:

[0014] If the feature correlation degree represented by the combined flight behavior analysis data does not meet the set correlation degree requirement, based on the flight behavior analysis data, the associated target flight feedback knowledge network is determined from the reference flight feedback knowledge network subsequence related to the flight interaction structured category included in the reference flight feedback knowledge network sequence.

[0015] In a possible implementation manner of the first aspect, determining an associated target flight feedback knowledge network from a reference flight feedback knowledge network sequence based on the combined flight behavior analysis data includes:

[0016] If the feature correlation degree represented by the combined flight behavior analysis data meets the set correlation degree requirement, the associated target flight feedback knowledge network is determined from the reference flight feedback knowledge network subsequence related to the flight interaction unstructured category of the flight interaction unstructured data included in the reference flight feedback knowledge network sequence.

[0017] In a possible implementation of the first aspect, the flight interaction unstructured data includes graph network data corresponding to a plurality of flight interaction unstructured categories, and the combined flight behavior analysis data is used to represent: a characteristic correlation degree between each of the plurality of flight interaction unstructured categories and the UAV flight monitoring data sequence;

[0018] Then, based on the combined flight behavior analysis data, determining an associated target flight feedback knowledge network from a reference flight feedback knowledge network sequence includes:

[0019] Based on the combined flight behavior analysis data, determining a target flight interaction unstructured category whose feature correlation meets a set correlation requirement from the plurality of flight interaction unstructured categories;

[0020] From a subsequence of reference flight feedback knowledge networks associated with the target flight interaction unstructured category, an associated target flight feedback knowledge network is determined.

[0021] In a possible implementation of the first aspect, performing combined flight behavior analysis on the UAV flight monitoring data sequence and the flight interaction unstructured data to generate corresponding combined flight behavior analysis data includes:

[0022] According to the set guidance strategy, based on the flight interaction structured data and the flight interaction unstructured data in the UAV flight monitoring data sequence, corresponding target guidance feature data is generated;

[0023] Using a combined flight behavior analysis network and based on a feature embedding representation strategy, feature embedding representation is performed on the flight interaction structured data in the target guidance feature data to generate a structured embedding representation vector;

[0024] Based on a target embedding representation strategy corresponding to a flight interaction unstructured category of the flight interaction unstructured data, feature embedding representation is performed on the flight interaction unstructured data to generate an unstructured embedding representation vector;

[0025] Integrating the structured embedding representation vector and the unstructured embedding representation vector to generate a target embedding representation vector;

[0026] Based on the target embedding representation vector, flight behavior analysis of multiple cyclic stages is performed to generate the combined flight behavior analysis result, wherein, in one cyclic flight behavior analysis stage, based on the target embedding representation vector and the analysis result of the previous cyclic stage, a combined flight behavior analysis result of this cyclic flight behavior analysis stage process is generated.

[0027] In a possible implementation of the first aspect, the generating corresponding target guidance feature data based on the flight interaction structured data and the flight interaction unstructured data in the UAV flight monitoring data sequence according to the set guidance strategy includes:

[0028] Based on the timing information between the at least one flight status monitoring cycle, constructing classification annotation data and interaction annotation data for the flight interaction structured data in the UAV flight monitoring data sequence, the classification annotation data is used to reflect: the corresponding flight interaction structured data is flight request interaction data or flight feedback report, and the interaction annotation data is used to reflect: the number of flight monitoring interactions corresponding to the corresponding flight interaction structured data;

[0029] Constructing corresponding category annotation data for the flight interaction unstructured data, wherein the category annotation data is used to reflect: the flight interaction unstructured category corresponding to the flight interaction unstructured data;

[0030] According to the data order set in the guidance strategy, each flight interaction structured data and flight interaction unstructured data after constructing the annotation data are integrated to generate the target guidance feature data.

[0031] In a possible implementation of the first aspect, the flight interaction unstructured data is flight status path diagram data;

[0032] Then, the target embedding representation strategy corresponding to the flight interaction unstructured category of the flight interaction unstructured data is used to perform feature embedding representation on the flight interaction unstructured data to generate an unstructured embedding representation vector, including:

[0033] Decomposing the flight state path diagram data to generate a plurality of flight state paths;

[0034] Performing graph convolution processing on the multiple flight state paths respectively to generate corresponding flight state graph convolution vectors;

[0035] generating corresponding node representation vectors based on the nodes of each of the plurality of flight state paths in the flight state path graph data;

[0036] Based on the generated node representation vectors, attention processing is performed on the generated flight state graph convolution vectors to generate an unstructured embedded representation vector of the flight state path graph data.

[0037] In a possible implementation of the first aspect, the training step of the combined flight behavior analysis network includes:

[0038] Obtaining a sample flight state monitoring sequence, wherein each sample flight state monitoring subsequence in the sample flight state monitoring sequence includes a UAV flight monitoring data sequence of at least one flight state monitoring cycle, wherein the UAV flight monitoring data sequence includes flight interaction structured data and flight interaction unstructured data, and each sample flight state monitoring subsequence carries a priori correlation between the UAV flight monitoring data sequence and the flight interaction unstructured data;

[0039] Based on the sample flight state monitoring sequence, parameter learning is performed on the deep learning network with initialized weight parameters until the network convergence requirements are met, wherein, in each training stage of the parameter learning, the UAV flight monitoring data sequence and the flight interaction unstructured data loaded into each sample flight state monitoring subsequence in this training stage are subjected to combined flight behavior analysis through the deep learning network scheduled in this training stage, and the estimated correlation between the UAV flight monitoring data sequence and the flight interaction unstructured data of each sample flight state monitoring subsequence is obtained respectively;

[0040] Based on the generated estimated correlations and the corresponding prior correlations, determine the network error parameters of the deep learning network scheduled in this training phase, and after updating the parameters of the deep learning network scheduled in this training phase based on the network error parameters, use the deep learning network with updated parameters to enter the next round of training phase;

[0041] In a possible implementation of the first aspect, the priori association degree is association degree label description information, the association degree label description information includes at least one description node, the estimated association degree includes a confidence sequence corresponding to the at least one description node, each confidence sequence includes a confidence that the corresponding description node is each reference description value in a reference description value sequence;

[0042] The determining of the network error parameters of the deep learning network scheduled in this training phase based on each generated estimated correlation degree and the corresponding prior correlation degree includes:

[0043] For an estimated association degree, determining a sub-error parameter corresponding to the estimated association degree according to the confidences corresponding to the actual description values ​​corresponding to the generated confidence sequences, wherein the actual description value is the description value of the corresponding description node in the association degree label description information;

[0044] The network error parameter is determined based on the sub-error parameters corresponding to the respective estimated correlation degrees.

[0045] In a possible implementation of the first aspect, based on the combined flight behavior analysis data, determining an associated target flight feedback knowledge network from a reference flight feedback knowledge network sequence includes:

[0046] Based on the feature correlation and the UAV flight monitoring data sequence, generating target flight interaction data of the most recent flight status monitoring cycle;

[0047] Determining, from the reference flight feedback knowledge network sequence, a reference flight feedback knowledge network subsequence associated with a plurality of flight interaction unstructured categories included in the target flight interaction data;

[0048] From the reference flight feedback knowledge network subsequence, an associated target flight feedback knowledge network is determined.

[0049] On the other hand, an embodiment of the present application also provides a deep learning-based UAV flight status monitoring system, including a processor and a machine-readable storage medium, wherein the machine-readable storage medium is connected to the processor, the machine-readable storage medium is used to store programs, instructions or codes, and the processor is used to execute the programs, instructions or codes in the machine-readable storage medium to implement the above method.

[0050] Based on the above aspects, the embodiment of the present application realizes a comprehensive and in-depth analysis of the UAV flight behavior by comprehensively using the structured and unstructured data in the UAV flight monitoring data sequence. Specifically, it is possible to accurately identify the characteristic correlation between the UAV flight monitoring data sequence and the flight interaction unstructured data, thereby effectively improving the accuracy and real-time performance of flight status monitoring. Furthermore, based on the combined flight behavior analysis data, it is possible to quickly lock the target flight feedback knowledge network that best matches the target flight behavior from the huge reference flight feedback knowledge network sequence, significantly enhancing the pertinence and effectiveness of the flight feedback. Finally, the flight request task feedback report generated using the target flight feedback knowledge network not only provides important guarantees for the safe flight of the UAV, but also provides a scientific basis for the optimization of subsequent flight strategies, greatly promoting the intelligent development of UAV technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1It is a schematic diagram of the execution flow of the UAV flight status monitoring method based on deep learning provided in an embodiment of the present application.

[0052] Figure 2 This is a schematic diagram of the hardware architecture of a drone flight status monitoring system based on deep learning provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] The present application will be described in detail below with reference to the accompanying drawings. Figure 1 This is a flow chart of a method for monitoring the flight status of a UAV based on deep learning provided by an embodiment of the present application. The method for monitoring the flight status of a UAV based on deep learning is introduced in detail below.

[0054] Step S110, obtaining a UAV flight monitoring data sequence of at least one flight status monitoring cycle, wherein the UAV flight monitoring data sequence at least includes: storing flight interaction structured data of a flight request task of a most recent flight status monitoring cycle.

[0055] In this embodiment, the server is first connected to the UAV flight data management system, and obtains the UAV flight monitoring data sequence of at least one flight status monitoring cycle through the UAV flight data management system. These UAV flight monitoring data sequences are collected and transmitted back in real time by the UAV during the flight, including various sensor data, flight status parameters, flight mission information, etc.

[0056] Specifically, the server retrieves the flight interaction structured data of the flight request task in the most recent flight status monitoring cycle from the data management system, and stores these flight interaction structured data in a structured format to facilitate subsequent processing and analysis by the server. The flight interaction structured data may include key information such as the start and end time of the flight task, flight route, flight altitude, speed, etc.

[0057] Step S120, when the UAV flight monitoring data sequence also includes flight interaction unstructured data, a combined flight behavior analysis is performed on the UAV flight monitoring data sequence and the flight interaction unstructured data to generate corresponding combined flight behavior analysis data, wherein the combined flight behavior analysis data is used to represent: the feature correlation between the UAV flight monitoring data sequence and the flight interaction unstructured data.

[0058] In this embodiment, after obtaining the UAV flight monitoring data sequence, the server checks whether the UAV flight monitoring data sequence also contains flight interaction unstructured data. Flight interaction unstructured data can exist in the form of images, videos, texts, etc., providing richer and more intuitive information about the flight process, but it is also relatively complex to process.

[0059] If the UAV flight monitoring data sequence does contain flight interaction unstructured data, the server will perform a combined flight behavior analysis on these flight interaction unstructured data. This analysis process aims to explore the feature correlation between the UAV flight monitoring data sequence and the flight interaction unstructured data.

[0060] Specifically, the server may use deep learning algorithms or machine learning models to jointly analyze flight interaction structured data and flight interaction unstructured data. Through analysis, the server can identify which flight interaction unstructured data is closely related to specific flight behaviors or states, thereby generating combined flight behavior analysis data.

[0061] Step S130, based on the combined flight behavior analysis data, determining an associated target flight feedback knowledge network from a reference flight feedback knowledge network sequence, wherein each reference flight feedback knowledge network in the reference flight feedback knowledge network sequence corresponds to a flight feedback knowledge point.

[0062] In this embodiment, based on the combined flight behavior analysis data, the server needs to determine the associated target flight feedback knowledge network from the reference flight feedback knowledge network sequence. The reference flight feedback knowledge network sequence is a pre-built knowledge base that contains various knowledge points related to flight feedback and corresponding network structures.

[0063] The server compares the combined flight behavior analysis data with each reference flight feedback knowledge network in the reference flight feedback knowledge network sequence to find the target flight feedback knowledge network that best matches or has the highest correlation with it. Once the target flight feedback knowledge network is determined, the server can use the flight feedback knowledge points contained in the target flight feedback knowledge network to gain a deeper understanding and analysis of the UAV's flight behavior.

[0064] Step S140, using the target flight feedback knowledge network, based on the UAV flight monitoring data sequence and the flight feedback knowledge points corresponding to the target flight feedback knowledge network, generating a flight feedback report for the flight request task.

[0065] Finally, the server uses the target flight feedback knowledge network to generate a flight feedback report for the flight request task based on the UAV flight monitoring data sequence and the flight feedback knowledge points corresponding to the target flight feedback knowledge network.

[0066] The flight feedback report is a comprehensive summary of the execution of the UAV flight mission. It includes key data during the flight, analysis results of flight behavior, and flight feedback and suggestions based on the reference knowledge network. This flight feedback report has important reference value for evaluating the execution effect of the flight mission, optimizing flight strategies, and improving flight safety.

[0067] In the specific implementation, the server first further parses and processes the UAV flight monitoring data sequence according to the knowledge points in the target flight feedback knowledge network. Then, the server can combine these processed data with the knowledge points to form a flight feedback report with a clear structure and rich content. Finally, the server saves this report and can send it to relevant users or systems as needed for further viewing and analysis.

[0068] In a detailed scenario example, assume that the server is processing flight monitoring data for a drone delivery mission. In step S110, the server obtains the flight interaction structured data of the most recent flight status monitoring cycle of the mission from the data management system, including take-off time, delivery location, flight route, flight speed, etc. In step S120, the server finds that these data sequences also contain flight interaction unstructured data, that is, a video shot by the drone during flight. The server uses a deep learning algorithm to analyze this video and identifies key frames and feature points, such as the take-off of the drone, obstacles during flight, and the scene of the delivery location. Then, the server combines these unstructured data with the flight interaction structured data for flight behavior analysis and generates combined flight behavior analysis data. In step S130, the server matches the combined flight behavior analysis data with the reference flight feedback knowledge network sequence. Through comparison and analysis, the server finds a target flight feedback knowledge network that best matches the target flight behavior, which contains flight feedback knowledge points about the drone delivery mission, such as flight route optimization, delivery location identification, etc. Finally, in step S140, the server uses the target flight feedback knowledge network to generate a flight feedback report for the flight request task based on the UAV flight monitoring data sequence and the flight feedback knowledge points corresponding to the target flight feedback knowledge network. This report records in detail the UAV's flight process, delivery status, and flight feedback and suggestions based on the reference knowledge network. For example, the report may point out that the UAV encountered some obstacles during the flight and recommends optimizing the flight route to avoid these obstacles; or the report may point out that the scene at the delivery location does not meet expectations and recommends further verification and investigation. This report has important reference value for evaluating the execution effect of the UAV delivery task, optimizing the flight strategy, and improving the delivery efficiency.

[0069] Based on the above steps, the embodiment of the present application realizes a comprehensive and in-depth analysis of the UAV flight behavior by comprehensively using the structured and unstructured data in the UAV flight monitoring data sequence. Specifically, it can accurately identify the characteristic correlation between the UAV flight monitoring data sequence and the flight interaction unstructured data, thereby effectively improving the accuracy and real-time performance of flight status monitoring. Furthermore, based on the combined flight behavior analysis data, it is possible to quickly lock the target flight feedback knowledge network that best matches the target flight behavior from the huge reference flight feedback knowledge network sequence, significantly enhancing the pertinence and effectiveness of the flight feedback. Finally, the flight request task feedback report generated by the target flight feedback knowledge network not only provides important guarantees for the safe flight of the UAV, but also provides a scientific basis for the optimization of subsequent flight strategies, greatly promoting the intelligent development of UAV technology.

[0070] In a possible implementation, after obtaining a UAV flight monitoring data sequence of at least one flight status monitoring cycle, the method further includes:

[0071] Step A110, based on the temporal correlation information in the UAV flight monitoring data sequence, the flight interaction structured data is state reconstructed to generate flight interaction structured data after state reconstruction.

[0072] Step A120, performing flight behavior analysis on the UAV flight monitoring data sequence and the flight interaction structured data after state reconstruction to generate corresponding flight behavior analysis data.

[0073] Then the step S130 includes: if the feature correlation degree represented by the combined flight behavior analysis data does not meet the set correlation degree requirement, based on the flight behavior analysis data, determining the associated target flight feedback knowledge network from the reference flight feedback knowledge network subsequence related to the flight interaction structured category included in the reference flight feedback knowledge network sequence.

[0074] After the server obtains the UAV flight monitoring data sequence, it notices that these UAV flight monitoring data sequences contain data from multiple flight status monitoring cycles, and there is a time series correlation between these data. In order to more accurately analyze flight behavior, the server decides to reconstruct the state of these flight interaction structured data.

[0075] In detail, the server first identifies the timestamp, flight phase mark and other information in the UAV flight monitoring data sequence. These time-series correlation information are used to describe the order and relationship between different flight status monitoring cycles. Then, using the extracted time-series correlation information, the server reorders and integrates the flight interaction structured data to generate flight interaction structured data after state reconstruction. In this process, the server may combine related data items according to the logical order of the flight mission to form a more complete and coherent description of the flight status.

[0076] After obtaining the flight interaction structured data after state reconstruction, the server next needs to perform flight behavior analysis on the UAV flight monitoring data sequence (including the reconstructed structured data) to generate flight behavior analysis data.

[0077] For example, the server integrates the UAV flight monitoring data sequence (including sensor data, flight state parameters, etc.) with the flight interaction structured data after state reconstruction to form a comprehensive flight data set. Next, the server analyzes this comprehensive flight data set using a preset flight behavior analysis model (which can be a model based on machine learning or deep learning). During the analysis process, the server can identify key events, abnormal behaviors, performance indicators, etc. during the flight process, and generate corresponding flight behavior analysis data.

[0078] After having the combined flight behavior analysis data (if there is flight interaction unstructured data and it has been analyzed) and the flight behavior analysis data, the server needs to determine the associated target flight feedback knowledge network from the reference flight feedback knowledge network sequence based on these data. If the feature correlation degree represented by the combined flight behavior analysis data does not meet the set correlation degree requirement, the server will make a determination based on the flight behavior analysis data.

[0079] The server first checks whether the feature correlation in the combined flight behavior analysis data (if available) meets the set correlation requirement, which can be a threshold set based on prior knowledge or business rules. If the feature correlation does not meet the requirement, the server will rely on the flight behavior analysis data instead. It searches the reference flight feedback knowledge network sequence for a subsequence of the reference flight feedback knowledge network that is related to the flight interaction structured category (after state reconstruction). This search process may be based on category labels, keyword matching, or more complex pattern recognition algorithms.

[0080] In the relevant reference flight feedback knowledge network subsequence, the server further compares the matching degree between the flight behavior analysis data and each reference flight feedback knowledge network, and finally determines the target flight feedback knowledge network with the highest correlation or the best match.

[0081] The above describes how, after obtaining the UAV flight monitoring data sequence, the target flight feedback knowledge network most relevant to the current flight behavior is finally determined through steps such as state reconstruction, flight behavior analysis, and knowledge network matching. This process not only makes full use of the real-time data during the UAV flight, but also combines prior knowledge and business rules to provide strong support for subsequent flight optimization, troubleshooting, etc.

[0082] In a possible implementation, step S130 may further include: if the feature correlation represented by the combined flight behavior analysis data meets the set correlation requirement, determining the associated target flight feedback knowledge network from a reference flight feedback knowledge network subsequence related to the flight interaction unstructured category of the flight interaction unstructured data included in the reference flight feedback knowledge network sequence.

[0083] In this embodiment, the server first checks whether the feature correlation represented in the combined flight behavior analysis data meets a preset correlation requirement, which may be a threshold set based on business rules, historical data experience or expert opinion.

[0084] For example, the server reads the correlation index in the combined flight behavior analysis data, which can be a value between 0 and 1, indicating the degree of correlation between the UAV flight monitoring data sequence and the flight interaction unstructured data. The server compares this correlation index with the set correlation threshold (for example, 0.8). If the correlation index is greater than or equal to 0.8, the server determines that the feature correlation meets the set requirements and proceeds to the next step; if it is less than 0.8, the target flight feedback knowledge network may be determined based on other logic (such as based on flight behavior analysis data), but this scenario does not involve this part.

[0085] Once it is determined that the feature correlation meets the set requirements, the server will next select subsequences related to the flight interaction unstructured data categories from the reference flight feedback knowledge network sequence.

[0086] For example, the server first identifies the specific categories of flight interaction unstructured data. For example, these data may include video data, image data, or text logs, and each category may have its own unique characteristics and processing methods. The server searches for subsequences related to these unstructured data categories in the reference flight feedback knowledge network sequence. The search process may be based on category labels, keyword matching, or pre-defined mapping relationships. For example, if the flight interaction unstructured data is video data, the server may look for all reference flight feedback knowledge network subsequences labeled "video analysis" or "visual feedback."

[0087] After finding the relevant reference flight feedback knowledge network subsequence, the server needs to further determine the target flight feedback knowledge network that best matches or is most relevant to the current flight behavior. For example, the server traverses each reference flight feedback knowledge network in the relevant subsequence, uses the combined flight behavior analysis data as input, and evaluates the matching degree of each reference flight feedback knowledge network with the current flight behavior. This evaluation process may involve complex pattern matching, similarity calculation, or rule-based logical reasoning.

[0088] The server may consider multiple factors, such as the flight feedback knowledge points contained in the reference flight feedback knowledge network, historical application effects, user evaluations, etc., to comprehensively determine which reference flight feedback knowledge network is most suitable for the current flight behavior analysis needs. Finally, the server determines the target flight feedback knowledge network with the highest correlation or the most consistent with the current flight behavior characteristics, and prepares to use the target flight feedback knowledge network to generate a flight feedback report.

[0089] The above describes how to select the target flight feedback knowledge network related to the flight interaction unstructured data category from the reference flight feedback knowledge network sequence step by step after judging that the feature correlation meets the set requirements based on the combined flight behavior analysis data. This process not only depends on the accuracy of the combined flight behavior analysis data, but also depends on the comprehensiveness and applicability of the reference flight feedback knowledge network, which provides strong support for the subsequent generation of flight feedback reports.

[0090] In a possible implementation, the flight interaction unstructured data includes graph network data corresponding to multiple flight interaction unstructured categories, and the combined flight behavior analysis data is used to represent: the feature correlation between each of the multiple flight interaction unstructured categories and the drone flight monitoring data sequence.

[0091] Then step S130 may further include:

[0092] Based on the combined flight behavior analysis data, a target flight interaction unstructured category whose feature correlation meets a set correlation requirement is determined from the multiple flight interaction unstructured categories.

[0093] From a subsequence of reference flight feedback knowledge networks associated with the target flight interaction unstructured category, an associated target flight feedback knowledge network is determined.

[0094] In this embodiment, the server first parses the combined flight behavior analysis data to understand the feature correlation between each flight interaction unstructured category and the UAV flight monitoring data sequence.

[0095] For example, the server reads the combined flight behavior analysis data, which may exist in the form of a table, matrix, or a specific data structure, where each row or item represents the association score between an unstructured flight interaction category and the UAV flight monitoring data sequence. The server traverses these data items and records the association score of each category. For example, the association score of the video category is 0.9, the association score of the image category is 0.7, and the association score of the text log category is 0.85.

[0096] Based on the combined flight behavior analysis data, the server determines the target category whose feature relevance meets the set relevance requirements from multiple flight interaction unstructured categories. For example, the server sets a relevance threshold (such as 0.8) to determine which categories have a high enough relevance to be the target of further analysis. The server compares the relevance score of each category with the threshold and finds that the relevance score of the video category, 0.9, exceeds the threshold, so it is determined as the target flight interaction unstructured category.

[0097] After determining the target flight interaction unstructured category, the server next selects subsequences related to the category from the reference flight feedback knowledge network sequence. For example, the server searches for all subsequences labeled "video analysis" or containing video processing related knowledge points in the reference flight feedback knowledge network sequence. These subsequences may contain specific feedback rules, optimization suggestions, or case studies for video data. The server may also need to further screen these subsequences to ensure that they are not only related to the video category, but also cover the key features or problem points in the current UAV flight monitoring data sequence.

[0098] After finding the relevant reference flight feedback knowledge network subsequences, the server needs to determine the target flight feedback knowledge network that best matches or is most relevant to the current flight behavior. For example, the server evaluates the flight feedback knowledge points in each relevant subsequence, considering their correlation with specific events, abnormal behaviors, or performance indicators in the UAV flight monitoring data sequence. Among them, the server may use some similarity calculation or pattern matching algorithm to quantify this correlation, and select the subsequence with the highest correlation as the target flight feedback knowledge network. The final target flight feedback knowledge network will be used to generate a detailed feedback report for the current flight behavior, including possible problem diagnosis, optimization suggestions, and improvement directions for future flight strategies.

[0099] Through the detailed description of the above scenario, it is described how to determine the target category with the required feature correlation based on the combined flight behavior analysis data when processing unstructured flight interaction data containing multiple categories, and further select the associated target flight feedback knowledge network from the reference flight feedback knowledge network sequence. This process not only improves the accuracy and pertinence of the analysis, but also provides strong support for the flight optimization and troubleshooting of UAVs.

[0100] In a possible implementation, step S120 includes:

[0101] Step S121, according to the set guidance strategy, based on the flight interaction structured data and the flight interaction unstructured data in the UAV flight monitoring data sequence, generate corresponding target guidance feature data.

[0102] In this embodiment, the server first generates target guidance feature data based on the set guidance strategy, combining the flight interaction structured data and the flight interaction unstructured data in the UAV flight monitoring data sequence. The guidance strategy may contain information such as data processing rules, order, weight distribution, etc., which is used to guide how to effectively integrate these two types of data. For example, the server extracts flight interaction structured data from the UAV flight monitoring data sequence, such as flight time, altitude, speed, route, etc. At the same time, the server identifies the categories of flight interaction unstructured data, such as video, image, text log, etc., and performs preliminary processing on them, such as video key frame extraction, image feature point detection, text keyword extraction, etc. According to the set guidance strategy, the server integrates these structured data and unstructured data according to certain rules to form target guidance feature data. For example, the strategy may require that the structured data at a specific time point be associated with its corresponding unstructured data (such as the video key frame at that time point) and assigned different weights.

[0103] Step S122, using the combined flight behavior analysis network, based on the feature embedding representation strategy, the flight interaction structured data in the target guidance feature data is feature embedded to generate a structured embedding representation vector.

[0104] In this embodiment, the server uses the feature embedding representation strategy in the combined flight behavior analysis network to perform feature embedding representation on the flight interaction structured data in the target guidance feature data and generate a structured embedding representation vector. This step aims to convert the structured data into a numerical vector form that is easier for the network to process.

[0105] For example, the server inputs the structured data in the target guidance feature data into the combined flight behavior analysis network. The feature embedding layer in the combined flight behavior analysis network uses a specific embedding matrix or function to convert each structured data item (such as flight time, altitude, etc.) into a fixed-length numerical vector, which is concatenated or superimposed to form a structured embedding representation vector that captures the key features of the structured data.

[0106] Step S123, based on the target embedding representation strategy corresponding to the flight interaction unstructured category of the flight interaction unstructured data, feature embedding representation is performed on the flight interaction unstructured data to generate an unstructured embedding representation vector.

[0107] In this embodiment, the server performs feature embedding representation on the unstructured data based on the target embedding representation strategy corresponding to the flight interaction unstructured data category to generate an unstructured embedding representation vector. Different unstructured data categories may require different embedding representation strategies.

[0108] For example, for video data, the server may use convolutional neural networks (CNN) to extract image features in key frames and convert these features into numerical vectors. For image data, the server can also use CNN or other image feature extraction methods (such as SIFT, SURF, etc.) to obtain image feature vectors. For text log data, the server may use natural language processing (NLP) technology, such as word embedding or sentence embedding, to convert text into numerical vectors. The feature vectors of these unstructured data are concatenated or weighted summed according to their respective categories to generate unstructured embedding representation vectors.

[0109] Step S124: Integrate the structured embedding representation vector and the unstructured embedding representation vector to generate a target embedding representation vector.

[0110] In this embodiment, the server integrates the structured embedded representation vector and the unstructured embedded representation vector to generate a target embedded representation vector. This step aims to fuse the features of the two types of data together to provide comprehensive input for subsequent flight behavior analysis.

[0111] For example, the server concatenates the structured embedding representation vector and the unstructured embedding representation vector to form a longer numerical vector. Alternatively, the server may use advanced fusion methods such as the Attention Mechanism to dynamically adjust the weights of the two types of data according to their importance and relevance, thereby generating a more representative target embedding representation vector.

[0112] Step S125, performing flight behavior analysis of multiple cyclic stages based on the target embedding representation vector to generate the combined flight behavior analysis result, wherein, in one cyclic flight behavior analysis stage, based on the target embedding representation vector and the analysis result of the previous cyclic stage, a combined flight behavior analysis result of this cyclic flight behavior analysis stage process is generated.

[0113] In this embodiment, the server performs flight behavior analysis in multiple loop stages based on the target embedding representation vector, and gradually generates a combined flight behavior analysis result. In each loop stage, the server uses the analysis result of the previous stage to optimize the analysis process of the current stage.

[0114] For example, the server inputs the target embedding representation vector into a recurrent neural network (RNN) or long short-term memory network (LSTM) or other recurrent layers in the combined flight behavior analysis network. In each cycle, the network updates its internal state based on the current target embedding representation vector and the analysis results of the previous stage (such as the hidden state vector), and generates the flight behavior analysis results of the current stage. These flight behavior analysis results may include flight behavior classification, anomaly detection, performance evaluation, etc., which will gradually become stable and accurate as the cycle progresses. Finally, the server integrates and optimizes the flight behavior analysis results of multiple cycle stages to generate the final combined flight behavior analysis results, which can be used for flight optimization, troubleshooting, performance evaluation, and other aspects of the drone.

[0115] In a possible implementation, step S121 includes:

[0116] Step S1211, based on the timing information between the at least one flight status monitoring cycle, construct classification annotation data and interaction annotation data for the flight interaction structured data in the UAV flight monitoring data sequence, the classification annotation data is used to reflect: the corresponding flight interaction structured data is flight request interaction data or flight feedback report, and the interaction annotation data is used to reflect: the number of flight monitoring interactions corresponding to the corresponding flight interaction structured data.

[0117] In this embodiment, the server first performs an in-depth analysis of the flight interaction structured data in the UAV flight monitoring data sequence based on the timing information between at least one flight status monitoring cycle, and constructs classification annotation data and interaction annotation data for it.

[0118] In detail, the server reads the UAV flight monitoring data sequence, identifies the timestamp of each flight status monitoring cycle, and the order between the cycles. For each flight interaction structured data item, the server determines whether it belongs to flight request interaction data or flight feedback report based on its content (such as message type, field information, etc.). For example, if the data item contains the words "takeoff request", it is classified as flight request interaction data; if it contains the words "flight status report", it is classified as flight feedback report. The server adds corresponding classification annotations to these data items, such as "request" or "feedback". In addition to the classification annotations, the server also counts the number of flight monitoring interactions corresponding to each flight interaction structured data item, which can be achieved by checking the interaction records, event counts, or specific fields in the data item. For example, if the data item records multiple communication attempts with the ground control station, the server will count the number of these attempts and add them to the corresponding data item as interaction annotation data.

[0119] Step S1212: construct corresponding category annotation data for the flight interaction unstructured data, where the category annotation data is used to reflect: the flight interaction unstructured category corresponding to the flight interaction unstructured data.

[0120] Next, the server constructs corresponding category annotation data for the flight interaction unstructured data to reflect the specific categories of these data (such as videos, images, text logs, etc.).

[0121] In detail, the server identifies the category of the flight interaction unstructured data by checking its format, file extension or content characteristics. For example, the server may identify files ending with ".mp4" as video data, files ending with ".jpg" as image data, and plain text files may be regarded as text logs. Once the category is identified, the server adds corresponding category annotations to each flight interaction unstructured data item. These annotations may exist in the form of tags, metadata or specific fields, and clearly indicate the category of the data.

[0122] Step S1213, integrating the various flight interaction structured data and flight interaction unstructured data after constructing the annotation data according to the data order set in the guidance strategy, to generate the target guidance feature data.

[0123] Finally, the server integrates the flight interaction structured data and unstructured data after constructing the annotation data according to the data order set in the guidance strategy to generate target guidance feature data.

[0124] In detail, the server first reads and parses the data order rules set in the guidance strategy, which may specify the order, priority, association method, etc. of data integration. For example, the rules may require the server to first process the flight request interaction data, and then integrate the relevant flight feedback reports and unstructured data in chronological order. According to the guidance strategy, the server integrates the annotated flight interaction structured data and unstructured data in the specified order, which may involve data sorting, alignment, association or merging operations. For example, the server may first arrange all flight request interaction data in chronological order and find the corresponding flight feedback report and unstructured data (such as video records when executing the request) for each request. Then, the server organizes these data items together in the format or structure specified in the guidance strategy to form target guidance feature data. The target guidance feature data finally generated can be a complex data structure, which contains the annotated flight interaction structured data and unstructured data, as well as the association information and temporal relationship between them. This data structure will be used as input for subsequent flight behavior analysis to help the server understand and analyze the flight behavior of the drone more accurately.

[0125] In a possible implementation, the flight interaction unstructured data is flight status path diagram data.

[0126] Then the step S123 includes:

[0127] Step S1231, decomposing the flight state path diagram data to generate multiple flight state paths.

[0128] In this embodiment, the server first receives the flight state path map data, which is a kind of unstructured graphic data that describes the state change path of the drone during flight. The server needs to decompose this data for subsequent processing.

[0129] In detail, the server receives flight state path graph data from the drone or related data storage system, and the flight state path graph data can exist in the form of a graph database, a file, or a data stream in a specific format. The server parses the flight state path graph data and identifies the nodes (representing different flight states) and edges (representing transitions or paths between states) in the graph. Based on the edge and node information in the graph, the server decomposes the flight state path graph data into multiple independent flight state paths. Each path represents the complete transition process of the drone from one state to another.

[0130] Step S1232: Perform graph convolution processing on the multiple flight state paths respectively to generate corresponding flight state graph convolution vectors.

[0131] The server then performs graph convolution on each decomposed flight state path. Graph convolution is a technique that applies convolution operations on graph structures to extract features of graph data.

[0132] In detail, the server selects a suitable graph convolutional network model, such as GCN (graph convolutional network), GraphSAGE, etc., which can learn the embedded representation of nodes based on the structure of the graph and the features of the nodes. The server takes each flight state path as the input of the graph convolutional network. Before input, it may be necessary to feature encode the nodes and edges in the path, such as using one-hot encoding, word embedding, or other feature extraction methods. The graph convolutional network updates the feature representation of the current node by aggregating the information of neighboring nodes. The server performs this operation to generate a new feature vector for each node, namely the flight state graph convolution vector.

[0133] Step S1233: Generate corresponding node representation vectors based on the nodes of each of the multiple flight state paths in the flight state path graph data.

[0134] In this embodiment, after obtaining the flight state graph convolution vector, the server also needs to generate corresponding node representation vectors based on the node information in the flight state path graph data. These node representation vectors will be used for subsequent attention processing.

[0135] In detail, the server may directly use the node features after graph convolution as the node representation vector, or further extract other features of the node (such as position, speed, height, etc.) according to specific requirements. If the node features are not in the form of numerical vectors, the server needs to encode and convert them. For example, convert the location information into a longitude and latitude coordinate vector, and convert the speed information into a numerical scalar.

[0136] Finally, the server generates a fixed-length numerical vector for each node as the node representation vector.

[0137] Step S1234, based on the generated node representation vectors, attention processing is performed on the generated flight state graph convolution vectors to generate an unstructured embedded representation vector of the flight state path graph data.

[0138] In this embodiment, the server uses the generated node representation vector to perform attention processing on the flight state graph convolution vector. The attention mechanism can help the model focus on the more important parts of the graph, thereby improving the accuracy and effectiveness of feature representation.

[0139] In detail, the server calculates the attention weight of each flight state graph convolution vector based on the node representation vector, which usually involves an attention scoring function that calculates the weight based on the similarity or correlation between the node representation vectors. The server uses the calculated attention weights to perform a weighted summation operation on the flight state graph convolution vector, so that important flight state paths and nodes will be more reflected in the final feature representation. After attention processing, the server splices or superimposes all the vectors obtained by weighted summation to form an unstructured embedded representation vector of the flight state path graph data, which captures the key features of the flight state path graph data and can be used for subsequent flight behavior analysis or other tasks.

[0140] In a possible implementation, the training step of the combined flight behavior analysis network includes:

[0141] Step S101, obtaining a sample flight status monitoring sequence, wherein each sample flight status monitoring subsequence in the sample flight status monitoring sequence includes a UAV flight monitoring data sequence of at least one flight status monitoring cycle, the UAV flight monitoring data sequence includes flight interaction structured data and flight interaction unstructured data, and each sample flight status monitoring subsequence carries a priori correlation between the UAV flight monitoring data sequence and the flight interaction unstructured data.

[0142] In this embodiment, the server first needs to obtain a large number of sample flight status monitoring sequences from a historical database or a simulation environment. These sample flight status monitoring sequences contain monitoring data of the UAV in different flight states for training the combined flight behavior analysis network.

[0143] The server is connected to the drone data management system and downloads monitoring data of different flight missions over a period of time. These data are organized into sample flight status monitoring sequences, each of which contains drone flight monitoring data for one or more flight status monitoring cycles. The server preprocesses the downloaded raw data, including data cleaning, format unification, feature extraction, etc. In particular, the server can ensure that each sample flight status monitoring subsequence contains flight interaction structured data and flight interaction unstructured data (such as flight status path diagram data). For each sample flight status monitoring subsequence, the server also needs to obtain or calculate the prior correlation between the drone flight monitoring data sequence and the flight interaction unstructured data, which is usually obtained through domain expert annotation, historical data analysis, or specific algorithm calculation.

[0144] Step S102: Based on the sample flight status monitoring sequence, parameter learning is performed on the deep learning network with initialized weight parameters until the network convergence requirements are met, wherein, in each training stage of the parameter learning, the UAV flight monitoring data sequence and the flight interaction unstructured data loaded into each sample flight status monitoring subsequence in this training stage are subjected to combined flight behavior analysis through the deep learning network scheduled in this training stage, and the estimated correlation between the UAV flight monitoring data sequence and the flight interaction unstructured data of each sample flight status monitoring subsequence is obtained respectively.

[0145] In this embodiment, the server uses a deep learning network with initialized weight parameters to iteratively train the sample flight status monitoring sequence, and optimizes its analysis capability of the combined flight behavior by continuously adjusting the network parameters.

[0146] The server loads a predefined deep learning network model that has a structure suitable for processing combined data (such as graph convolution layers, attention mechanisms, etc.). The server initializes the network's weight parameters to random values ​​or pre-trained values. During the training process, the server loads sample flight status monitoring subsequences into the network in a certain order or randomly. Each training stage may only process a portion of the subsequences to speed up training and reduce memory usage. For each subsequence loaded into the network, the server performs combined flight behavior analysis, which includes feature embedding representations of flight interaction structured data and unstructured data, respectively, and then integrating them for comprehensive analysis. The network outputs the estimated correlation between the drone flight monitoring data sequence and the flight interaction unstructured data for each subsequence.

[0147] Step S103, based on the generated estimated correlations and the corresponding prior correlations, determine the network error parameters of the deep learning network scheduled for this training stage, and after updating the parameters of the deep learning network scheduled for this training stage based on the network error parameters, use the deep learning network with updated parameters to enter the next round of training.

[0148] At the end of each training phase, the server can calculate the difference between the estimated correlation output by the network and the prior correlation in the sample data, that is, the network error. The server then adjusts the weight parameters of the network based on this error, hoping to reduce the error in the next round of training.

[0149] The server traverses all processed sample flight status monitoring subsequences and calculates the error between the estimated correlation and the prior correlation of each subsequence, which is usually achieved through a loss function (such as mean square error, cross entropy, etc.). Based on the calculated total error, the server uses a back-propagation algorithm and optimizer (such as Adam, SGD, etc.) to update the weight parameters of the network. This process aims to minimize the error and improve the prediction accuracy of the network. After each parameter update, the server can check whether the network has converged, which is usually judged by observing the changing trend of the network error parameters, the performance on the validation set, or reaching the preset number of training rounds. If the network has converged or reached the stopping condition, the training process ends; otherwise, the server will continue to use the updated network parameters to enter the next round of training.

[0150] Through the above steps, the server can gradually train a combined flight behavior analysis network that can accurately analyze the correlation between the UAV flight monitoring data sequence and the flight interaction unstructured data. This combined flight behavior analysis network can be used in practical applications to generate flight feedback reports, optimize flight strategies, and improve flight safety.

[0151] In one possible implementation, the prior association is association label description information, the association label description information includes at least one description node, the estimated association includes a confidence sequence corresponding to the at least one description node, and each confidence sequence contains the confidence that the corresponding description node is each reference description value in the reference description value sequence.

[0152] Step S103 includes:

[0153] Step S1031, for an estimated association degree, determine the sub-error parameter corresponding to the estimated association degree according to the confidences corresponding to the actual description values ​​corresponding to the generated confidence sequences, and the actual description value is the description value of the corresponding description node in the association degree label description information.

[0154] Step S1032: determining the network error parameter based on the sub-error parameters corresponding to the respective estimated correlation degrees.

[0155] In this embodiment, the server is training a combined flight behavior analysis network, which is designed to analyze the correlation between the UAV flight monitoring data sequence and the flight interaction unstructured data. The prior correlation is given in the form of correlation label description information, which includes multiple description nodes and their corresponding actual description values ​​(i.e., reference description values). The estimated correlation output by the combined flight behavior analysis network includes the confidence sequence corresponding to each of these description nodes, and each element in the confidence sequence represents the confidence that the description node is a specific value in the reference description value sequence.

[0156] The server first parses the data structure of the prior association and estimated association to clarify the relationship and correspondence between them. In detail, the server reads the association tag description information, identifies the description nodes (such as "flight altitude stability", "path complexity", etc.) and the actual description values ​​(such as "high", "medium", "low" for "flight altitude stability", "complex", "normal", "simple" for "path complexity"). The server understands the estimated association structure output by the network, and knows that each description node corresponds to a confidence sequence, and each element in the sequence corresponds to the confidence of a reference description value.

[0157] For each estimated relevance, the server needs to calculate its corresponding sub-error parameter, which usually involves comparing the confidence in the estimated relevance with the actual description value. In detail, the server traverses each description node in the estimated relevance and its confidence sequence, and finds the confidence corresponding to the actual description value (i.e., the description value of the corresponding description node in the relevance label description information) in the confidence sequence. The server uses a loss function (such as cross entropy loss) to calculate the difference between the confidence corresponding to the actual description value and the ideal confidence (usually 1, indicating complete certainty), which is the sub-error parameter of the description node.

[0158] After all the sub-error parameters of the estimated association are calculated, the server needs to aggregate them to obtain the error parameters of the entire network. In detail, according to actual needs, the server may need to weight the sub-errors of different description nodes to reflect their different importance in the overall association evaluation. The server adds all the weighted sub-error parameters (or merges them according to other aggregation rules) to obtain the network error parameters of the deep learning network scheduled in this training phase.

[0159] After obtaining the network error parameters, the server will use them to guide the weight update process of the deep learning network. For example, the server uses the back propagation algorithm to propagate the network error parameters from the output layer to the input layer layer by layer, and calculate the gradient of each layer parameter. Based on the calculated gradient, the server uses an optimization algorithm (such as Adam, SGD, etc.) to update the network weight parameters in the hope of reducing the error in the next round of training.

[0160] Through the above steps, the server can accurately calculate the error of the deep learning network during the training process, and use this error to guide the network parameter update, thereby gradually optimizing the network's ability to analyze the correlation between the UAV flight monitoring data sequence and the flight interaction unstructured data.

[0161] In a possible implementation, step S130 further includes:

[0162] Step S131, based on the feature correlation and the UAV flight monitoring data sequence, generating the target flight interaction data of the most recent flight status monitoring cycle.

[0163] Step S132: determining, from the reference flight feedback knowledge network sequence, a reference flight feedback knowledge network subsequence associated with a plurality of flight interaction unstructured categories included in the target flight interaction data.

[0164] Step S133: determining an associated target flight feedback knowledge network from the reference flight feedback knowledge network subsequence.

[0165] In this embodiment, the server has completed the combined flight behavior analysis, generated analysis data including feature correlation, and has a sequence of multiple reference flight feedback knowledge networks covering different flight interaction unstructured categories and flight feedback knowledge points. The goal of the server is to find the target flight feedback knowledge network that is most relevant to the most recent flight status monitoring cycle.

[0166] The server first generates the target flight interaction data of the most recent flight status monitoring cycle based on the feature correlation and the UAV flight monitoring data sequence. These data will serve as the basis for the subsequent search for the target flight feedback knowledge network. In detail, the server integrates the UAV flight monitoring data sequence within the most recent flight status monitoring cycle, including the feature correlation in the flight interaction structured data (such as flight time, altitude, speed, etc.) and the combined flight behavior analysis data. Based on the feature correlation, the server extracts features with a high correlation with the flight interaction unstructured data, which may point to specific flight behaviors, abnormal events or critical paths. Combining the extracted features with the UAV flight monitoring data, the server generates the target flight interaction data, which not only contains the specific values ​​of the flight status, but also incorporates the feature information closely related to the flight interaction unstructured data.

[0167] The server then selects reference flight feedback knowledge network subsequences associated with multiple flight interaction unstructured categories included in the target flight interaction data from the reference flight feedback knowledge network sequence. In detail, the server first identifies the flight interaction unstructured categories (such as videos, images, text logs, etc.) involved in the target flight interaction data. Then, the server traverses the reference flight feedback knowledge network sequence to find reference flight feedback knowledge networks containing these categories, which are selected to form one or more subsequences, each of which is associated with at least one unstructured category in the target flight interaction data.

[0168] After obtaining the associated reference flight feedback knowledge network subsequences, the server needs to further determine the target flight feedback knowledge network from these subsequences that is most relevant to the target flight interaction data and can provide the most useful feedback. In detail, the server conducts a comprehensive evaluation of each candidate reference flight feedback knowledge network. The evaluation may be based on multiple factors, such as the historical application effect of the network, user evaluation, and feature matching degree with the target flight interaction data. Based on the evaluation results, the server prioritizes the candidate networks. The sorting may use a complex algorithm or a simple scoring mechanism. Finally, the server selects the reference flight feedback knowledge network with the highest priority as the target flight feedback knowledge network, which will be used to generate a flight feedback report for the most recent flight status monitoring cycle.

[0169] Through the above steps, the server can accurately determine the associated target flight feedback knowledge network from the reference flight feedback knowledge network sequence based on the combined flight behavior analysis data, providing strong support for the subsequent flight feedback report generation.

[0170] Figure 2 The hardware structure of the UAV flight status monitoring system 100 based on deep learning for implementing the UAV flight status monitoring method based on deep learning provided in the embodiment of the present application is shown as follows: Figure 2 As shown, the deep learning-based drone flight status monitoring system 100 may include a processor 110 , a machine-readable storage medium 120 , a bus 130 , and a communication unit 140 .

[0171] In one possible design, the UAV flight state monitoring system 100 based on deep learning can be a single server or a server group. The server group can be centralized or distributed (for example, the UAV flight state monitoring system 100 based on deep learning can be a distributed system). In some embodiments, the UAV flight state monitoring system 100 based on deep learning can be local or remote. For example, the UAV flight state monitoring system 100 based on deep learning can access information and / or data stored in the machine-readable storage medium 120 via a network. For another example, the UAV flight state monitoring system 100 based on deep learning can be directly connected to the machine-readable storage medium 120 to access the stored information and / or data. In some embodiments, the UAV flight state monitoring system 100 based on deep learning can be implemented on a UAV flight state monitoring system based on deep learning. As an example only, the UAV flight state monitoring system based on deep learning can include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an internal cloud, a multi-layer cloud, etc. or any combination thereof.

[0172] The machine-readable storage medium 120 may store data and / or instructions. In some embodiments, the machine-readable storage medium 120 may store data acquired from an external terminal. In some embodiments, the machine-readable storage medium 120 may store data and / or instructions that the deep learning-based drone flight status monitoring system 100 uses to execute or use to complete the exemplary method described in this application.

[0173] During the specific implementation process, one or more processors 110 execute computer executable instructions stored in the machine-readable storage medium 120, so that the processor 110 can execute the drone flight status monitoring method based on deep learning in the above method embodiment. The processor 110, the machine-readable storage medium 120 and the communication unit 140 are connected through the bus 130, and the processor 110 can be used to control the sending and receiving actions of the communication unit 140.

[0174] The specific implementation process of the processor 110 can refer to the various method embodiments executed by the above-mentioned deep learning-based UAV flight status monitoring system 100. The implementation principles and technical effects are similar, and this embodiment will not be repeated here.

[0175] In addition, an embodiment of the present application also provides a readable storage medium, in which computer executable instructions are preset. When the processor executes the computer executable instructions, the above-mentioned drone flight status monitoring method based on deep learning is implemented.

[0176] It should be noted that in order to simplify the description disclosed in the present application and thereby facilitate the understanding of one or more embodiments of the invention, in the foregoing description of the embodiments of the present application, various features are sometimes grouped into one embodiment, drawing or description thereof.

Claims

1. A method for monitoring the flight status of an unmanned aerial vehicle based on deep learning, characterized in that: The method comprises: Obtaining a UAV flight monitoring data sequence of at least one flight status monitoring cycle, wherein the UAV flight monitoring data sequence at least includes: storing flight interaction structured data of a flight request task of a most recent flight status monitoring cycle; When the UAV flight monitoring data sequence also includes flight interaction unstructured data, performing a combined flight behavior analysis on the UAV flight monitoring data sequence and the flight interaction unstructured data to generate corresponding combined flight behavior analysis data, wherein the combined flight behavior analysis data is used to represent: a feature correlation between the UAV flight monitoring data sequence and the flight interaction unstructured data; Based on the combined flight behavior analysis data, determining an associated target flight feedback knowledge network from a reference flight feedback knowledge network sequence, each reference flight feedback knowledge network in the reference flight feedback knowledge network sequence corresponding to a flight feedback knowledge point; Using the target flight feedback knowledge network, based on the UAV flight monitoring data sequence and the flight feedback knowledge points corresponding to the target flight feedback knowledge network, a flight feedback report of the flight request task is generated; The step of determining an associated target flight feedback knowledge network from a reference flight feedback knowledge network sequence based on the combined flight behavior analysis data comprises: If the feature correlation degree represented by the combined flight behavior analysis data meets the set correlation degree requirement, determining the associated target flight feedback knowledge network from the reference flight feedback knowledge network subsequence related to the flight interaction unstructured category of the flight interaction unstructured data included in the reference flight feedback knowledge network sequence; The flight interaction unstructured data includes graph network data corresponding to a plurality of flight interaction unstructured categories respectively, and the combined flight behavior analysis data is used to represent: the feature correlation between each of the plurality of flight interaction unstructured categories and the UAV flight monitoring data sequence; Then, based on the combined flight behavior analysis data, determining an associated target flight feedback knowledge network from a reference flight feedback knowledge network sequence includes: Based on the combined flight behavior analysis data, determining a target flight interaction unstructured category whose feature correlation meets a set correlation requirement from the plurality of flight interaction unstructured categories; Determining, from a subsequence of reference flight feedback knowledge networks associated with the target flight interaction unstructured category, an associated target flight feedback knowledge network; The performing combined flight behavior analysis on the UAV flight monitoring data sequence and the flight interaction unstructured data to generate corresponding combined flight behavior analysis data includes: According to the set guidance strategy, based on the flight interaction structured data and the flight interaction unstructured data in the UAV flight monitoring data sequence, corresponding target guidance feature data is generated; Using a combined flight behavior analysis network and based on a feature embedding representation strategy, feature embedding representation is performed on the flight interaction structured data in the target guidance feature data to generate a structured embedding representation vector; Based on a target embedding representation strategy corresponding to a flight interaction unstructured category of the flight interaction unstructured data, feature embedding representation is performed on the flight interaction unstructured data to generate an unstructured embedding representation vector; Integrating the structured embedding representation vector and the unstructured embedding representation vector to generate a target embedding representation vector; Based on the target embedding representation vector, flight behavior analysis of multiple cyclic stages is performed to generate the combined flight behavior analysis result, wherein, in one cyclic flight behavior analysis stage, based on the target embedding representation vector and the analysis result of the previous cyclic stage, a combined flight behavior analysis result of this cyclic flight behavior analysis stage process is generated.

2. The method for monitoring the flight status of a UAV based on deep learning according to claim 1, characterized in that: After obtaining the UAV flight monitoring data sequence of at least one flight status monitoring cycle, the method further includes: Based on the time series correlation information in the UAV flight monitoring data sequence, the flight interaction structured data is state reconstructed to generate flight interaction structured data after state reconstructing; Performing flight behavior analysis on the UAV flight monitoring data sequence and the flight interaction structured data after state reconstruction to generate corresponding flight behavior analysis data; Then, based on the combined flight behavior analysis data, determining an associated target flight feedback knowledge network from a reference flight feedback knowledge network sequence includes: If the feature correlation degree represented by the combined flight behavior analysis data does not meet the set correlation degree requirement, based on the flight behavior analysis data, the associated target flight feedback knowledge network is determined from the reference flight feedback knowledge network subsequence related to the flight interaction structured category included in the reference flight feedback knowledge network sequence.

3. The method for monitoring the flight status of a UAV based on deep learning according to claim 1, characterized in that: The method generates corresponding target guidance feature data based on the set guidance strategy and the flight interaction structured data and the flight interaction unstructured data in the UAV flight monitoring data sequence, including: Based on the timing information between the at least one flight status monitoring cycle, constructing classification annotation data and interaction annotation data for the flight interaction structured data in the UAV flight monitoring data sequence, the classification annotation data is used to reflect: the corresponding flight interaction structured data is flight request interaction data or flight feedback report, and the interaction annotation data is used to reflect: the number of flight monitoring interactions corresponding to the corresponding flight interaction structured data; Constructing corresponding category annotation data for the flight interaction unstructured data, wherein the category annotation data is used to reflect: the flight interaction unstructured category corresponding to the flight interaction unstructured data; According to the data order set in the guidance strategy, each flight interaction structured data and flight interaction unstructured data after constructing the annotation data are integrated to generate the target guidance feature data.

4. The method for monitoring the flight status of a UAV based on deep learning according to claim 1, characterized in that: The flight interaction unstructured data is flight status path diagram data; Then, the target embedding representation strategy corresponding to the flight interaction unstructured category of the flight interaction unstructured data is used to perform feature embedding representation on the flight interaction unstructured data to generate an unstructured embedding representation vector, including: Decomposing the flight state path diagram data to generate a plurality of flight state paths; Performing graph convolution processing on the multiple flight state paths respectively to generate corresponding flight state graph convolution vectors; generating corresponding node representation vectors based on the nodes of each of the plurality of flight state paths in the flight state path graph data; Based on the generated node representation vectors, attention processing is performed on the generated flight state graph convolution vectors to generate an unstructured embedded representation vector of the flight state path graph data.

5. The method for monitoring the flight status of a UAV based on deep learning according to claim 3, characterized in that: The training steps of the combined flight behavior analysis network include: Obtaining a sample flight state monitoring sequence, wherein each sample flight state monitoring subsequence in the sample flight state monitoring sequence includes a UAV flight monitoring data sequence of at least one flight state monitoring cycle, wherein the UAV flight monitoring data sequence includes flight interaction structured data and flight interaction unstructured data, and each sample flight state monitoring subsequence carries a priori correlation between the UAV flight monitoring data sequence and the flight interaction unstructured data; Based on the sample flight state monitoring sequence, parameter learning is performed on the deep learning network with initialized weight parameters until the network convergence requirements are met, wherein, in each training stage of the parameter learning, the UAV flight monitoring data sequence and the flight interaction unstructured data loaded into each sample flight state monitoring subsequence in this training stage are subjected to combined flight behavior analysis through the deep learning network scheduled in this training stage, and the estimated correlation between the UAV flight monitoring data sequence and the flight interaction unstructured data of each sample flight state monitoring subsequence is obtained respectively; Based on the generated estimated correlations and the corresponding prior correlations, determine the network error parameters of the deep learning network scheduled in this training phase, and after updating the parameters of the deep learning network scheduled in this training phase based on the network error parameters, use the deep learning network with updated parameters to enter the next round of training phase; The priori association degree is association degree label description information, the association degree label description information includes at least one description node, the estimated association degree includes a confidence sequence corresponding to the at least one description node, and each confidence sequence includes the confidence that the corresponding description node is each reference description value in the reference description value sequence; The determining of the network error parameters of the deep learning network scheduled in this training phase based on each generated estimated correlation degree and the corresponding prior correlation degree includes: For an estimated association degree, determining a sub-error parameter corresponding to the estimated association degree according to the confidences corresponding to the actual description values ​​corresponding to the generated confidence sequences, wherein the actual description value is the description value of the corresponding description node in the association degree label description information; The network error parameter is determined based on the sub-error parameters corresponding to the respective estimated correlation degrees.

6. The method for monitoring the flight status of a UAV based on deep learning according to any one of claims 1 to 2, characterized in that: Based on the combined flight behavior analysis data, an associated target flight feedback knowledge network is determined from the reference flight feedback knowledge network sequence, including: Based on the feature correlation and the UAV flight monitoring data sequence, generating target flight interaction data of the most recent flight status monitoring cycle; Determining, from the reference flight feedback knowledge network sequence, a reference flight feedback knowledge network subsequence associated with a plurality of flight interaction unstructured categories included in the target flight interaction data; From the reference flight feedback knowledge network subsequence, an associated target flight feedback knowledge network is determined.

7. A UAV flight status monitoring system based on deep learning, characterized in that: The deep learning-based UAV flight status monitoring system includes a processor and a memory, the memory is connected to the processor, the memory is used to store programs, instructions or codes, and the processor is used to execute the programs, instructions or codes in the memory to implement the deep learning-based UAV flight status monitoring method described in any one of claims 1 to 6 above.

Images