Unmanned aerial vehicle inspection method and equipment based on multi-modal data analysis and medium

By collecting and fusing audio and video data on a drone, generating feature sequence data sets and inputting multimodal detection models, the problem of one-sided and low robustness of behavioral detection results in the prior art is solved, and more accurate and flexible behavioral analysis.

CN119048951APending Publication Date: 2024-11-29ZHUHAI ZHONGKE HUIZHI TECH CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411033340.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-30
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

Existing behavior detection schemes usually only analyze data from a single source, and the results are one-sided and low robust, and cannot effectively reflect the behavioral patterns of the target object.

Method used

Using the UAV patrol method based on multimodal data analysis, the UAV collects audio data and video data for preprocessing, and fuses them to generate feature sequence data sets, and enters the multimodal detection model to obtain behavioral detection results. . The method includes extracting audio features using the Mel frequency cepspectral coefficients, DETR neural network model generates the region of interest in the video frame, and weighted splicing of audio and video features through a cross attention mechanism.

Benefits of technology

It improves the accuracy and robustness of behavior detection, can be applied to various scenarios, meets diverse behavior detection needs, and provides richer behavior information and contextual environment perception.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119048951A_ABST
    Figure CN119048951A_ABST
Patent Text Reader

Abstract

The invention discloses an unmanned aerial vehicle inspection method and device based on multi-modal data analysis and a medium. The method comprises the following steps: collecting audio data and video data of a target object through an unmanned aerial vehicle; preprocessing the audio data and the video data; fusing the audio data and the video data to generate a feature sequence data set; and inputting the feature sequence data set into a multi-modal detection model to obtain a behavior detection result of the target object. According to the embodiment of the invention, the detection model is constructed by adopting image and voice multi-modal data input, and various sensor data are fused, so that richer behavior information and context environment information can be obtained, and the detection depth and accuracy are improved. According to the embodiment of the invention, behavior analysis is carried out through multi-modal fusion, so that the robustness and adaptability of the method can be effectively improved, and the method can be suitable for various scenes and meet diversified behavior detection requirements.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal data analysis, and in particular to a drone inspection method, equipment and medium based on multimodal data analysis. Background Art

[0002] With the rapid development of smart home, smart transportation, smart city and other fields, people's demand for more intelligent, automated and personalized services is gradually increasing. Behavior detection of target objects refers to the use of computer vision and artificial intelligence technology to identify and understand the behavior patterns of specific targets in video or image sequences. Behavior detection can effectively assist service providers in behavioral analysis and decision-making for different scenarios, so that service providers can optimize target objects and provide more targeted services to more efficiently meet the needs of people and society.

[0003] However, existing behavior detection schemes usually only perform behavior analysis on data from a single source. The analysis results are one-sided and have low robustness, and therefore cannot effectively reflect the behavior patterns of the target objects. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a drone inspection method, device and medium based on multimodal data analysis.

[0005] like Figure 1 As shown, the first aspect of the present invention provides a drone inspection method based on multimodal data analysis, comprising the following steps:

[0006] Collect audio and video data of target objects through drones;

[0007] Preprocessing audio data and video data;

[0008] Fusion of the audio data and the video data to generate a feature sequence data set;

[0009] The feature sequence data set is input into a multimodal detection model to obtain a behavior detection result of the target object.

[0010] Furthermore, the video data is acquired by a camera mounted on a drone; the video data is multi-angle visual information of the target object;

[0011] The audio data is obtained through a microphone carried by the drone; the audio data is audio information emitted by the target object.

[0012] Furthermore, in the step of preprocessing the audio data and the video data, the audio data is preprocessed, specifically comprising the following steps:

[0013] Extracting features of the audio data by Mel-frequency cepstral coefficients to obtain MFCC features of the audio data;

[0014] The video data is preprocessed, specifically including the following steps:

[0015] Dividing the video data into a plurality of video frames;

[0016] The DETR neural network model is used to generate the region of interest for each video frame in the video data.

[0017] Furthermore, the feature extraction of the audio data by Mel-frequency cepstral coefficients to obtain the MFCC features of the audio data specifically includes the following steps:

[0018] Performing FFT transformation on the audio data to convert the audio data from a time domain signal into a frequency domain signal to obtain frequency domain audio data;

[0019] Taking the power spectrum of the frequency domain audio data, inputting it into the Mel filter bank, and obtaining the FBank audio feature;

[0020] The FBank audio features are compressed using discrete cosine transform to obtain the MFCC features of the audio data.

[0021] Furthermore, the fusion of the audio data and the video data to generate a feature sequence data set specifically includes the following steps:

[0022] The MFCC feature vector extracted from the audio data at each moment is intra-frame associated with the region of interest of the video frame to obtain an associated result;

[0023] Converting the video frame into an image feature matrix, and flattening the image feature matrix to obtain a one-dimensional image extraction feature vector;

[0024] According to the association result, weighted splicing is performed on the MFCC feature vector extracted from the audio data and the image feature obtained by converting the video frame to obtain a plurality of splicing features;

[0025] The splicing features are combined to obtain a feature sequence data set.

[0026] Furthermore, the MFCC features of the audio data and the image features converted from the video frames are weightedly spliced ​​to obtain a plurality of splicing features, and the weighted splicing is specifically performed using the following formula:

[0027] w=[p1,p 2....... ]

[0028] feature_v1=k(n)

[0029] feature_v2=k(m) ......

[0031] final_vector=w[0]*feature_v1+w[1]*feature_v2+......

[0032] Among them, w represents a preset splicing weight ratio, p1 and p2 represent the ratio distribution probability, m represents the MFCC feature vector, n represents the image extraction feature vector; 1, 2... represent the splicing sequence determined according to the association result.

[0033] Furthermore, the multimodal detection model includes a Liner neural network module, a Resnet residual dense convolutional layer module and a cross-attention mechanism CrossAttention module;

[0034] The Liner neural network module is used to perform feature learning of audio data on the feature sequence data set to obtain an audio feature sequence;

[0035] The dense convolutional layer module of the Resnet residual is used for the feature sequence data set to perform feature learning of video data to obtain a video feature sequence;

[0036] The cross-attention mechanism CrossAttention module is used to perform fitting learning on the audio feature sequence and the video feature sequence to obtain the target object behavior detection result.

[0037] Furthermore, the Liner neural network module specifically includes a first Liner module, a gated convolution combination module, a Relu activation function module and a second Liner module;

[0038] The first Liner module is used to perform a linearization operation on the feature sequence data set;

[0039] The gated convolution combination module includes three learning layers with the same structure; the input of the first learning layer is a feature sequence data set for linearization operation, and the following steps are performed in each learning layer:

[0040] The input of the learning layer is used for feature learning through the following formula:

[0041] g = tanh(w f,k *x)·σ(w q,k *x)

[0042] Among them, * represents the convolution operation, tanh and σ represent two different activation function mechanisms, and w f,k and w q,krepresents the learnable weight parameters of the two activation function mechanisms for the feature sequence data set, represents the multiplication operation symbol at the corresponding position, and g is the output first audio feature intermediate sequence;

[0043] After obtaining the first audio feature intermediate sequence, use a 1*1 convolution module to reduce the dimension of the audio feature sequence to obtain a second audio feature intermediate sequence; input the second audio feature intermediate sequence into a Relu activation function module;

[0044] Concatenate the second audio feature intermediate sequence with the feature sequence data set of the linearization operation of the input learning layer to obtain a third audio feature intermediate sequence; and use the third audio feature intermediate sequence as input to the next learning layer;

[0045] The Relu activation function module receives the second audio feature intermediate sequences output by the three learning layers, and obtains three fourth audio feature intermediate sequences after the Relu activation function operation;

[0046] The second Liner module performs fitting learning on the three fourth audio feature intermediate sequences to obtain a final audio feature sequence;

[0047] The Relu activation function module receives the third audio feature intermediate sequences output by the three learning layers, and obtains three fourth audio feature intermediate sequences after the Relu activation function operation;

[0048] The second Liner module performs fitting learning on the three fourth audio feature intermediate sequences to obtain a final audio feature sequence;

[0049] The dense convolutional layer module of the Resnet residual includes three dense convolutional layers and a BiLstm module with the same structure;

[0050] Each dense convolutional layer performs convolution feature extraction on the feature sequence data set to obtain an intermediate sequence of video features as the output of each dense convolutional layer;

[0051] The BiLstm module is used to summarize the intermediate sequences of video features output by the three dense convolutional layers for fitting learning to obtain the final video feature sequence;

[0052] The CrossAttention module of the cross attention mechanism is used to receive the audio feature sequence output by the Liner neural network module and the video feature sequence output by the dense convolution layer module of the Resnet residual, and then perform fitting learning on the audio feature sequence and the video feature sequence. After completing the fitting learning, the target object behavior detection result in the form of probability distribution is obtained through Sigmoid activation function operation.

[0053] A second aspect of the present invention provides an electronic device, including a processor and a memory;

[0054] The memory is used to store programs;

[0055] The processor executes the program to implement the UAV inspection method based on multimodal data analysis.

[0056] A third aspect of the present invention provides a computer-readable storage medium, which stores a program, and the program is executed by a processor to implement the drone inspection method based on multimodal data analysis.

[0057] The embodiment of the present invention also discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the above method.

[0058] The embodiments of the present invention have the following beneficial effects: The embodiments of the present invention use two multimodal data inputs, image and voice, to construct a detection model, fuse multiple sensor data, and can obtain richer behavior information and contextual environment information, thereby improving the depth and accuracy of detection. The embodiments of the present invention perform behavior analysis through multimodal fusion, which can effectively improve the robustness and adaptability of the method, so that the method of the present invention can be applied to various scenarios and meet diverse behavior detection needs.

[0059] Additional aspects and advantages of the present invention will be given in part in the following description, and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0061] Figure 1 It is a basic implementation flow chart of a drone inspection method based on multimodal data analysis of the present invention;

[0062] Figure 2 It is a schematic diagram of the processing flow of the multimodal detection model of the present invention;

[0063] Figure 3 It is a schematic diagram of the structure of an electronic device of the present invention;

[0064] Figure 4 It is a schematic diagram of the structure of a computer-readable storage medium of the present invention. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0066] With the continuous development of sensor technology, the functions and performance of various sensors have been greatly improved, including cameras, microphones, etc. These sensors can collect more abundant data, providing a better foundation for multimodal behavior detection based on automatic inspection of drones. Artificial intelligence technologies such as machine learning and deep learning have made great progress in the past few decades. These technologies are able to process large-scale multimodal data, extract and learn characteristic patterns in the data, and build efficient and accurate behavior detection models. This provides powerful behavior analysis capabilities for the behavior detection system based on automatic inspection of drones.

[0067] In view of this, the present invention proposes a multimodal behavior detection method based on automatic inspection of drones. Compared with the traditional behavior detection method which only uses a single voice or image data input, the present invention uses two multimodal data inputs, image and voice, to construct a detection model.

[0068] like Figure 1 As shown, an embodiment of the present invention discloses a drone inspection method based on multimodal data analysis, comprising the following steps:

[0069] S1. Collect audio and video data of the target object through drones;

[0070] S2. Preprocessing the audio data and the video data;

[0071] S3. Fusing audio data and video data to generate a feature sequence data set;

[0072] S4. Input the feature sequence dataset into the multimodal detection model to obtain the behavior detection results of the target object.

[0073] The implementation process of each step of the present invention is discussed in detail below.

[0074] S1. Collect audio and video data of the target object through the drone.

[0075] In the embodiment of the present invention, the video data of the target object is acquired by a camera mounted on the drone, and the acquired video data is a real-time video image stream, which can acquire multi-angle visual information of the target object. Acquiring multi-angle visual information of the target object can help solve the problem of incomplete target coverage under a single viewing angle and improve the robustness of multimodal analysis.

[0076] The audio data of the target object is obtained through the microphone on the drone, which can capture the audio data of the target object. The audio data of the target object can further supplement the deficiency of the video data, thereby providing more comprehensive target object behavior data.

[0077] S2. Preprocess the audio data and video data.

[0078] In the embodiment of the present invention, the preprocessing of the audio data is mainly performed by extracting MFCC features from the audio data through Mel-frequency cepstral coefficients.

[0079] Specifically, in order to process the audio data of each segment, especially the audio segment corresponding to the ROI area generated by the real-time image of the camera, and better analyze the information of the audio data, the present invention performs Fourier transform FFT on the audio signal to convert the audio signal from the time domain to the frequency domain. The following is the calculation formula:

[0080]

[0081] Where F(k) represents the kth discrete frequency in the frequency domain, x(n) represents the nth sampling point of the time domain signal, N represents the total number of samplings of the signal, j is an imaginary unit, and m is a positive integer. The present invention takes N=512 to calculate the FFT of 512 points. It can be understood as a complex rotation factor, which represents the amplitude information of the phase.

[0082] In MFCC, FBank is usually used to replace the output of the Mel filter bank. The power spectrum of the speech signal is passed through the Mel filter bank to obtain the Fbank feature calculation formula as follows:

[0083]

[0084] Among them, S(m) indicates that the result of the output FBank is a vector, in which each element represents the logarithmic value of the output energy of a Mel filter, x a (k) represents the power spectrum of the signal, H mIndicates the center amplitude of the filter in the filter bank, k indicates the number of each filter in the filter bank k∈[1,+∞), and Ln is the natural logarithm. Usually in MFCC, discrete cosine transform (DCT) is further applied to compress FBank to obtain the MFCC coefficients. The following is the calculation formula:

[0085]

[0086] Wherein Mc(n) is the output MFCC feature, which is usually 12-16 dimensional. The present invention takes 13 dimensional features. S(m) represents the FBank feature vector, where M is the number of triangular filters, and n is the MFCC coefficient order n∈[1,2,......].

[0087] On the other hand, the preprocessing of video data is mainly achieved through the DETR (Detection Transformer) neural network model. First, the video data is divided into multiple video frames, and then the DETR neural network model is used to generate the ROI (Region of Interest) for the real-time image.

[0088] The process of generating ROI area based on DETR neural network model can be summarized as follows:

[0089] Feature extraction: Use CNN (such as ResNet, etc.) to extract features from video frames.

[0090] Position encoding: Add position encoding to the extracted feature map to help the model understand the spatial position of the feature.

[0091] Transformer encoding and decoding: The feature map is fed into the Transformer encoder-decoder, the encoder enhances the features and captures global context information; a fixed number of trends (queries) in the decoder interact with the encoded features and gradually correct the match between the queries and the targets in the image.

[0092] Prediction: Finally, the decoder outputs predicted bounding box coordinates and class probabilities for each query.

[0093] Post-processing: Perform post-processing steps such as non-maximum suppression (NMS) on the output bounding boxes and categories to remove overlapping bounding boxes and retain the most likely detection results as the final ROI area.

[0094] S3. Fuse the audio data and video data to generate a feature sequence dataset.

[0095] The fusion of audio and video data first requires the establishment of data association. Specifically, the MFCC feature vector extracted from the audio data at each moment and the region of interest of the video frame are subjected to intra-frame association processing to obtain the association result. Secondly, the video frame is converted into an image feature matrix. This matrix is ​​composed of multiple vectors, and each vector represents certain information in the image, such as edges, textures, etc. This matrix is ​​flattened into a one-dimensional image extraction feature vector, and this vector can be used to represent the characteristics of the entire image. For example, this vector can be regarded as splitting the image from left to right and from top to bottom into several blocks and arranging them in order. In this way, the image information of the present invention is converted into a vector representation and weightedly spliced ​​with the audio feature MFCC feature sequence. The following is the calculation formula:

[0096] w=[p1,p 2....... ]

[0097] feature_v1=k(n)

[0098] feature_v2=k(m) ......

[0100] final_vector=w[0]*feature_v1+w[1]*feature_v2+......

[0101] Where w represents the splicing weight ratio p1, p2 is the proportional allocation probability, feature_v1=k(n) and feature_v2=k(m) represent the image extraction feature vector and audio extraction MFCC feature vector n∈(1,+∞), m∈(1,+∞), respectively, and final_vector represents the feature sequence data set after weighted allocation.

[0102] S4. Input the feature sequence dataset into the multimodal detection model to obtain the behavior detection results of the target object.

[0103] Step S4 is mainly implemented through a multimodal detection model. Figure 2 As shown, the multimodal detection model of the embodiment of the present invention mainly includes a Liner neural network module, a dense convolutional layer module of Resnet residuals, and a CrossAttention mechanism module. The Liner neural network module is used to perform feature learning of audio data on a feature sequence data set to obtain an audio feature sequence; the dense convolutional layer module of Resnet residuals is used to perform feature learning of video data on a feature sequence data set to obtain a video feature sequence; the CrossAttention mechanism module is used to perform fitting learning on audio feature sequences and video feature sequences to obtain target object behavior detection results.

[0104] Specifically, the Liner neural network module of the present invention includes a first Liner module, a gated convolution combination module, a Relu activation function module, and a second Liner module;

[0105] The first Liner module is used to perform linearization operations on the feature sequence data set.

[0106] The gated convolution combination module includes three learning layers with the same structure; the input of the first learning layer is the feature sequence data set of the linearization operation, and the following steps are performed in each learning layer:

[0107] The input of the learning layer is used for feature learning through the following formula:

[0108] g = tanh(w f,k *x)·σ(w q,k *x)

[0109] Among them, * represents the convolution operation, tanh and σ represent two different activation function mechanisms, and w f,k and w q,k represents the learnable weight parameters of the two activation function mechanisms for the feature sequence data set, represents the multiplication operation symbol at the corresponding position, and g is the output first audio feature intermediate sequence;

[0110] After obtaining the first audio feature intermediate sequence, use a 1*1 convolution module to reduce the dimension of the audio feature sequence to obtain a second audio feature intermediate sequence; input the second audio feature intermediate sequence into a Relu activation function module;

[0111] The second audio feature intermediate sequence is concatenated with the feature sequence data set of the linearization operation of the input learning layer to obtain a third audio feature intermediate sequence; and the third audio feature intermediate sequence is used as the input of the next learning layer;

[0112] The Relu activation function module receives the second audio feature intermediate sequences output by the three learning layers, and obtains three fourth audio feature intermediate sequences after the Relu activation function operation;

[0113] The second Liner module performs fitting learning on the three fourth audio feature intermediate sequences to obtain a final audio feature sequence.

[0114] Since the embodiment of the present invention uses three learning layers to learn the feature sequence data set, the formed fourth audio feature intermediate sequence has a residual structure, which can better complete the fitting learning.

[0115] The Relu activation function module receives the third audio feature intermediate sequences output by the three learning layers, and obtains three fourth audio feature intermediate sequences after the Relu activation function operation;

[0116] The second Liner module performs fitting learning on the three fourth audio feature intermediate sequences to obtain a final audio feature sequence.

[0117] The dense convolutional layer module of Resnet residual includes three dense convolutional layers with the same structure and a BiLstm module;

[0118] Each dense convolutional layer performs convolution feature extraction on the feature sequence data set, and obtains the intermediate sequence of video features as the output of each dense convolutional layer;

[0119] The BiLstm module is used to summarize the intermediate sequences of video features output by the three dense convolutional layers for fitting learning to obtain the final video feature sequence.

[0120] The CrossAttention module is used to receive the audio feature sequence output by the Liner neural network module and the video feature sequence output by the Resnet residual dense convolutional layer module, and then perform fitting learning on the audio feature sequence and the video feature sequence. After completing the fitting learning, the Sigmoid activation function is used to obtain the target object behavior detection result in the form of probability distribution.

[0121] The UAV inspection method based on multimodal data analysis of the present invention has the following advantages:

[0122] 1. Multimodal data fusion: Using multiple sensor data such as images and voice simultaneously can obtain richer behavioral information and improve detection accuracy.

[0123] 2. Contextual awareness: Fusion of multiple sensor data can better perceive the contextual environment information of behavior and enhance understanding of behavior.

[0124] 3. Strong anti-interference ability: A single sensor is easily affected by environmental interference, while multimodal fusion can improve the robustness of the system.

[0125] 4. Diversity and flexibility: This method is applicable to various scenarios, meets diverse behavior detection needs, and has strong adaptability.

[0126] The present invention also proposes a method for annotating a training set based on a DETR neural network model by integrating speech and real-time camera images. The neural network model fitted by the training set annotated by this method has good applicability. The model is trained based on a data set fused with speech data and camera images, where speech is collected by an onboard microphone and images are collected by an onboard camera. A decision-making judgment model is finally formed after multimodal data preprocessing at the front end and a cross-attention mechanism. The parallelism of the two data interfaces makes the multimodal detection model more accurate and efficient in application.

[0127] Figure 3 : is a schematic diagram of the structure of the electronic device proposed in the second embodiment of the present invention. In this embodiment, the memory stores program instructions for implementing the drone inspection method based on multimodal data analysis of any of the above embodiments. The processor is used to execute the program instructions stored in the memory to perform drone inspection based on multimodal data analysis. Among them, the processor can also be called CPU (Central Processing Unit). The processor may be an integrated circuit chip with signal processing capabilities. The processor can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0128] The contents of the method in the first embodiment of the present invention are all applicable to the electronic device embodiment. The functions specifically implemented by the electronic device embodiment are the same as those of the above method embodiment, and the beneficial effects achieved are also the same as those achieved by the above method.

[0129] Figure 4 It is a schematic diagram of the structure of the computer-readable storage medium of the third embodiment of the present invention. The computer-readable storage medium of the fourth embodiment of the present invention stores program instructions that can implement the above-mentioned drone inspection method based on multimodal data analysis, wherein the program instructions can be stored in the above-mentioned storage medium in the form of a software product, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each embodiment method of the present invention. The aforementioned computer-readable storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes, or terminal devices such as computers, servers, mobile phones, tablets, etc.

[0130] The contents of the method in the first embodiment of the present invention are all applicable to the computer-readable storage medium embodiment. The functions specifically implemented by the computer-readable storage medium embodiment are the same as those of the above method embodiment, and the beneficial effects achieved are also the same as those achieved by the above method.

[0131] This embodiment also provides a computer program product. When the computer program product runs on a computer, it enables the computer to execute the above-mentioned related steps to implement the drone inspection method based on multimodal data analysis provided in the above embodiment.

[0132] Those skilled in the art will appreciate that the modules in the devices in the embodiments of the present invention may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments of the present invention may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including corresponding claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including corresponding claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0133] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0134] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0135] In addition, each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. In particular, for embodiments such as devices and equipment, since they are basically similar to method embodiments, the relevant parts can refer to the partial description of the method embodiments. The embodiments of the devices and equipment described above are merely schematic, wherein the modules, units, etc. described as separate components may or may not be physically separated, that is, they may be located in one place, or they may be distributed in multiple places, such as nodes of a system network. Specifically, some or all of the modules and units may be selected according to actual needs to achieve the purpose of the above-mentioned embodiment scheme. Those skilled in the art can understand and implement it without paying creative labor.

[0136] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0137] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0138] In addition, the terms "first", "second", etc. used in the embodiments of the present invention are only used for descriptive purposes and should not be understood as indicating or implying relative importance, or implicitly indicating the number of technical features indicated in the present embodiment. Therefore, the features defined by the terms "first", "second", etc. in the embodiments of the present invention can explicitly or implicitly indicate that the embodiment includes at least one of the features. In the description of the present invention, the word "multiple" means at least two or two or more, such as two, three, four, etc., unless otherwise clearly and specifically defined in the embodiments.

[0139] In the embodiments of the present invention, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element. In addition, components, features, and elements with the same name in different embodiments of the present invention may have the same meaning or different meanings, and their specific meanings need to be determined by their explanation in the specific embodiment or further combined with the context in the specific embodiment.

[0140] Although embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and cannot be construed as limitations of the present invention, and those of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present invention. Those skilled in the art will readily come to think of other embodiments of the present invention after considering the specification and practicing the present invention. This application is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art that are not disclosed in the present invention. The specification and embodiments are intended to be exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.

Claims

1. A drone inspection method based on multimodal data analysis, characterized in that: The following steps are involved: Collect audio and video data of target objects through drones; Preprocessing audio data and video data; Fusion of the audio data and the video data to generate a feature sequence data set; The feature sequence data set is input into a multimodal detection model to obtain a behavior detection result of the target object.

2. The drone inspection method based on multimodal data analysis according to claim 1 is characterized in that: The video data is obtained through a camera mounted on the drone; the video data is multi-angle visual information of the target object; The audio data is obtained through a microphone carried by the drone; the audio data is audio information emitted by the target object.

3. The drone inspection method based on multimodal data analysis according to claim 1 is characterized in that: In the step of preprocessing the audio data and the video data, the audio data is preprocessed, specifically comprising the following steps: Extracting features of the audio data by Mel-frequency cepstral coefficients to obtain MFCC features of the audio data; The video data is preprocessed, specifically including the following steps: Dividing the video data into a plurality of video frames; The DETR neural network model is used to generate the region of interest for each video frame in the video data.

4. The drone inspection method based on multimodal data analysis according to claim 3 is characterized in that: The feature extraction of the audio data by Mel-frequency cepstral coefficients to obtain the MFCC features of the audio data specifically includes the following steps: Performing FFT transformation on the audio data to convert the audio data from a time domain signal into a frequency domain signal to obtain frequency domain audio data; Taking the power spectrum of the frequency domain audio data, inputting it into the Mel filter bank, and obtaining the FBank audio feature; The FBank audio features are compressed using discrete cosine transform to obtain the MFCC features of the audio data.

5. The drone inspection method based on multimodal data analysis according to claim 1 is characterized in that: The step of fusing the audio data and the video data to generate a feature sequence data set specifically includes the following steps: The MFCC feature vector extracted from the audio data at each moment is intra-frame associated with the region of interest of the video frame to obtain an associated result; Converting the video frame into an image feature matrix, and flattening the image feature matrix to obtain a one-dimensional image extraction feature vector; According to the association result, weighted splicing is performed on the MFCC feature vector extracted from the audio data and the image feature obtained by converting the video frame to obtain a plurality of splicing features; The splicing features are combined to obtain a feature sequence data set.

6. The drone inspection method based on multimodal data analysis according to claim 5 is characterized in that: The MFCC features of the audio data and the image features converted from the video frame are weightedly spliced ​​to obtain a plurality of splicing features, and the weighted splicing is specifically performed using the following formula: in=[p1,p 2....... ] feature_v1=k(n) feature_v2=k(m) ...... final_vector=w[0]*feature_v1+w[1]*feature_v2+...... Among them, w represents a preset splicing weight ratio, p1 and p2 represent the ratio distribution probability, m represents the MFCC feature vector, n represents the image extraction feature vector; 1, 2... represent the splicing sequence determined according to the association result.

7. The drone inspection method based on multimodal data analysis according to claim 1 is characterized in that: The multimodal detection model includes a Liner neural network module, a Resnet residual dense convolutional layer module and a cross-attention mechanism CrossAttention module; The Liner neural network module is used to perform feature learning of audio data on the feature sequence data set to obtain an audio feature sequence; The dense convolutional layer module of the Resnet residual is used for the feature sequence data set to perform feature learning of video data to obtain a video feature sequence; The cross-attention mechanism CrossAttention module is used to perform fitting learning on the audio feature sequence and the video feature sequence to obtain the target object behavior detection result.

8. The drone inspection method based on multimodal data analysis according to claim 7 is characterized in that: The Liner neural network module specifically includes a first Liner module, a gated convolution combination module, a Relu activation function module, and a second Liner module; The first Liner module is used to perform a linearization operation on the feature sequence data set; The gated convolution combination module includes three learning layers with the same structure; the input of the first learning layer is a feature sequence data set for linearization operation, and the following steps are performed in each learning layer: The input of the learning layer is used for feature learning through the following formula: g=tanh(w f,k *x)·σ(w q,k *x) Among them, * represents the convolution operation, tanh and σ represent two different activation function mechanisms, and w f,k and w q,k represents the learnable weight parameters of the two activation function mechanisms for the feature sequence data set, represents the multiplication operation symbol at the corresponding position, and g is the output first audio feature intermediate sequence; After obtaining the first audio feature intermediate sequence, use a 1*1 convolution module to reduce the dimension of the audio feature sequence to obtain a second audio feature intermediate sequence; input the second audio feature intermediate sequence into a Relu activation function module; Concatenate the second audio feature intermediate sequence with the feature sequence data set of the linearization operation of the input learning layer to obtain a third audio feature intermediate sequence; and use the third audio feature intermediate sequence as input to the next learning layer; The Relu activation function module receives the second audio feature intermediate sequences output by the three learning layers, and obtains three fourth audio feature intermediate sequences after the Relu activation function operation; The second Liner module performs fitting learning on the three fourth audio feature intermediate sequences to obtain a final audio feature sequence; The dense convolutional layer module of the Resnet residual includes three dense convolutional layers and a BiLstm module with the same structure; Each dense convolution layer performs convolution feature extraction on the feature sequence data set to obtain an intermediate sequence of video features as the output of each dense convolution layer; The BiLstm module is used to summarize the intermediate sequences of video features output by the three dense convolutional layers for fitting learning to obtain the final video feature sequence; The CrossAttention module of the cross attention mechanism is used to receive the audio feature sequence output by the Liner neural network module and the video feature sequence output by the dense convolution layer module of the Resnet residual, and then perform fitting learning on the audio feature sequence and the video feature sequence. After completing the fitting learning, the target object behavior detection result in the form of probability distribution is obtained through Sigmoid activation function operation.

9. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement a drone inspection method based on multimodal data analysis as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that: The storage medium stores a program, and the program is executed by a processor to implement a drone inspection method based on multimodal data analysis as described in any one of claims 1-8.

Citation Information

Cited By

  • Behavior detection method and device, electronic equipment and storage medium

    CN121659230A