Unmanned aerial vehicle image classification model generation method based on big and small model fusion

Abstract

The invention discloses an unmanned aerial vehicle image classification model generation method based on big and small model fusion, and the method comprises the following steps: firstly, carrying out the preprocessing of data cleaning, positive and negative sampling and the like on the data collected by an unmanned aerial vehicle; designing a low-dimensional feature extraction module in the large model to extract low-dimensional Embedding, and ensuring the consistency of the dimension and the size of the Embedding; secondly, designing a high-dimensional feature extraction module to extract high-dimensional Embedding, and enabling the Embedding to have higher expression and discrimination capabilities through networks such as an attention mechanism and a linear network; the high-dimensional Embedding is input into a super network, and the super network uses the high-dimensional Embedding and the preprocessed data to generate an image classification small model for each unmanned aerial vehicle; and finally, through a picture classification result of the small model and the large model, analyzing the operation situation of the unmanned aerial vehicle and generating countermeasures. The method has the beneficial effects that the accuracy of unmanned aerial vehicle picture classification is improved, and professional countermeasures can be provided.

Description

Technical Field

[0001] The present invention relates to a large model, a fusion of large and small models, a model generation method, and a staged feature vector generation, and in particular to a large and small model fusion unmanned aerial vehicle image classification model generation method. Background Art

[0002] In the UAV image classification task, the rapid development of deep learning technology has provided a broad application space for intelligent image analysis. In the face of the time delay, data security and privacy protection problems encountered by the traditional data centralized processing mode in the UAV image classification task, federated learning technology has become an effective solution.

[0003] There are still two problems when using federated learning for distributed training in drone image classification: first, different drones have different missions, which poses the challenge of non-independent and identically distributed (Non-IID) data; second, this small neural network lacks model knowledge and question-answering capabilities.

[0004] In order to solve the problem of non-independent and identically distributed data, personalized federated learning optimizes data privacy protection and model generalization ability by generating small models through hypernetworks; in order to further improve the performance of drone image classification tasks, the application of large language models (LLMs) is introduced to enhance the understanding of the deep meaning of images. In the drone swarm image classification task, the strategy of combining large and small models achieves the complementarity of the text understanding ability of large models and the efficient computing advantages of small models, significantly improving the efficiency and safety of drone reconnaissance tasks.

[0005] Currently, no effective solution has been proposed for the problems in the related technologies. Summary of the invention

[0006] In view of the problems in the related technologies, the present invention proposes a method for generating a drone image classification model by integrating large and small models, which is characterized by overcoming the above technical problems existing in the existing related technologies. To this end, the specific technical solutions adopted by the present invention are as follows:

[0007] A method for generating a drone image classification model by integrating large and small models, characterized in that the method comprises the following steps:

[0008] S1. Design a large and small model architecture, which includes a server and multiple user nodes. The large model, super network and other modules are deployed on the server side, responsible for user question and answer, and the generation of small model for image classification. The small model for image classification is deployed on the drone node, responsible for real-time information collection, processing and training;

[0009] S2. The drone is equipped with hardware such as a high-definition camera, GPS, and a microcontroller. An information acquisition module and a data preprocessing module are set in the microcontroller. The information acquisition module uses a high-definition camera and GPS to collect pictures and location information in real time. The data preprocessing module performs preprocessing operations such as data cleaning on the collected data.

[0010] S3. Design low-dimensional feature extraction modules and high-dimensional feature extraction modules on the large model side, use the embedding network to extract low-dimensional features (Embedding) from the preprocessed image, and perform feature alignment after generating low-dimensional Embedding, including scale normalization, direction adjustment, etc., to ensure the consistency of their dimensions and sizes in subsequent processing;

[0011] S4. Input the aligned low-dimensional embeddings into the high-dimensional feature extractor. Inside the high-dimensional feature extractor, the low-dimensional embeddings are transformed into high-dimensional embeddings through a series of transformations and reasoning operations such as self-attention mechanism and feedforward neural network. It has stronger expression and discrimination capabilities and can more comprehensively describe the complex content and structure in the image.

[0012] S5. The large model uses the high-dimensional embedding as the input of the super network. Under the guidance of the high-dimensional embedding, the super network combines the image data of each drone for joint training to generate a small personalized image classification model for each drone.

[0013] S6. In the inference stage, each drone uses a local personalized image classification model to classify the collected images and sends the classification results to the large model. In the decision-making stage, the large model performs final situation analysis and decision-making on the drone based on the original images and classification results.

[0014] Furthermore, the data preprocessing module in S2 performs preprocessing operations such as data cleaning on the collected operation data;

[0015] First, the drone dataset is divided into training set, validation set and test set according to the preset ratio. Then, in the data preprocessing module, including image scaling, normalization, data enhancement, etc., the generalization ability of the model is improved, oversampling and undersampling are used to balance the imbalance of positive and negative sample data, and feature selection and polynomial feature generation are performed at the same time.

[0016] Furthermore, in S3, the low-dimensional feature extraction of the preprocessed image is performed using an embedding network, and the feature alignment process after generating the low-dimensional embedding includes the following steps:

[0017] After preprocessing the data, the embedding network E in the large model is used to extract features to obtain a low-dimensional Embedding e = E (I), and then the mean μ is calculated through e e With standard deviation σ e , normalize e;

[0018] After the normalization operation, the angle θ between the current feature vector and the target direction is adjusted according to the set target direction vector d to achieve the alignment of the low-dimensional Embedding and the high-dimensional Embedding.

[0019] Furthermore, the low-dimensional embedding in S4 is converted into a high-dimensional embedding through a series of transformations and inference operations such as a self-attention mechanism and a feedforward neural network, including the following steps:

[0020] First, use e aligned Calculate the attention query Q, K, V, calculate the attention weight Attention, and calculate the weighted value A; then, use the linear transformation and activation function to feed forward A h ffn =ReLU(AW1+b1)W2+b2; Finally, all transformed outputs are combined to generate a high-dimensional Embedding e h .

[0021] Furthermore, in S5, under the guidance of high-dimensional embedding, the super network is combined with the image data of each drone for joint training, and generating a personalized image classification model for each drone also includes the following steps:

[0022] On the server side, the hypernetwork first determines the number of layers and the number of neurons in each layer to generate parameters based on the target network; then, the hypernetwork is updated using the gradient descent method; during the hypernetwork update process, each node inputs the parameters into the local network for forward and backward propagation and updates the parameters. Calculate the change in the hypernetwork distribution parameters And upload it to the server; finally, the hypernetwork iteratively updates the network parameters according to the parameter changes uploaded by each node until the hypernetwork training process is completed after convergence.

[0023] Furthermore, in the inference stage of S6, each drone uses a local personalized image classification model to classify the collected pictures, and sends the classification results to the large model. In the decision stage, the large model performs a final situation analysis and decision on the drone based on the original pictures and the classification results; the steps include:

[0024] Generate personalized model parameters θ for each client in the hypernetwork iAfter that, when the drone takes a new picture, it first uses the personalized model to get the classification result of the picture and sends it to the big model;

[0025] The large model then combines the image with the classification results to analyze the situation of the drone in the current image and plan the next response measures.

[0026] The beneficial effects of the present invention are:

[0027] The present invention proposes a method for generating a drone image classification model that integrates large and small models, and designs a new method for drone image classification that integrates large and small models. This paper designs a low-dimensional feature extraction module in the large model to ensure the dimension and size consistency of the Embedding; designs a high-dimensional feature extraction module, and uses networks such as the attention mechanism to make the Embedding have stronger expression and discrimination capabilities; then inputs the high-dimensional Embedding into the hypernetwork to generate a small image classification model; finally, through the classification results of the small model, the large model analyzes the operation status of the drone and generates countermeasures; the combination of large and small models can combine the strong text processing capabilities of the large model and the advantages of the small model in processing complex data. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0029] Figure 1 A flowchart for the collaborative completion of drone image classification by large and small models of the present invention;

[0030] Figure 2 This is a flow chart of the super network training of the present invention; DETAILED DESCRIPTION

[0031] To further illustrate each embodiment, the present invention provides drawings, which are part of the disclosure of the present invention and are mainly used to illustrate the embodiments and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these contents, ordinary technicians in the field should be able to understand other possible implementations and advantages of the present invention. The components in the figures are not drawn to scale, and similar component symbols are generally used to represent similar components.

[0032] According to an embodiment of the present invention, a method for generating a drone image classification model by integrating large and small models is provided. First, the drone collects image information, and the data preprocessing module cleans and preprocesses the data; then, a low-dimensional feature extraction module is designed in the large model to ensure the dimension and size consistency of the Embedding; a high-dimensional feature extraction module is designed, and the Embedding has stronger expression and discrimination capabilities through networks such as the attention mechanism; the high-dimensional Embedding is then input into the hypernetwork to generate a small image classification model; finally, through the classification results of the small model, the large model analyzes the operation status of the drone and generates countermeasures.

[0033] The present invention is further described with reference to the accompanying drawings and specific embodiments. Figure 1 As shown, according to an embodiment of the present invention, a method for generating a drone image classification model by integrating large and small models comprises the following steps:

[0034] Step S1: Design a large and small model architecture, which includes a server and multiple user nodes. The large model, super network and other modules are deployed on the server side, which is responsible for user question and answer and the generation of small model for image classification. The small model for image classification is deployed on the drone node, which is responsible for real-time information collection, processing and training.

[0035] Step S2: The drone is equipped with hardware such as a high-definition camera, GPS, and a microcontroller. An information acquisition module and a data preprocessing module are set in the microcontroller. The information acquisition module uses the high-definition camera and GPS to collect pictures and location information in real time. The data preprocessing module performs preprocessing operations such as data cleaning on the collected data.

[0036] The following steps are also included:

[0037] [1] The drone dataset is divided into training set, validation set and test set according to the preset ratio;

[0038] [2] In the data preprocessing module, including image scaling, normalization, data enhancement, etc., the generalization ability of the model is improved. Oversampling and undersampling are used to balance the imbalance of positive and negative sample data. Feature selection and polynomial feature generation are also performed.

[0039] Step S3: Design low-dimensional feature extraction modules and high-dimensional feature extraction modules on the large model side, use the embedding network to extract low-dimensional features from the preprocessed image, generate low-dimensional embeddings, and perform feature alignment processing, including scale normalization, direction adjustment, etc., to ensure the consistency of their dimensions and sizes in subsequent processing;

[0040] The following steps are also included:

[0041] [1] After preprocessing the data, the embedding network E in the large model is used to extract features to obtain a low-dimensional Embedding e = E(I), and then the mean μ is calculated through e e With standard deviation σ e , normalize e

[0042] [2] After the normalization operation, the angle θ between the current feature vector and the target direction is adjusted according to the set target direction vector d. aligned =R(θ)·e norm , Achieve alignment between low-dimensional embeddings and high-dimensional embeddings.

[0043] Step S4: Input the aligned low-dimensional embeddings into the high-dimensional feature extractor. Inside the high-dimensional feature extractor, the low-dimensional embeddings are transformed into high-dimensional embeddings through a series of transformations and inference operations such as self-attention mechanism and feedforward neural network. It has stronger expression and discrimination capabilities and can more comprehensively describe the complex content and structure in the image.

[0044] The following steps are also included:

[0045] [1] Use e aligned Calculate the attention query Q = e aligned W Q , K = e alianed W K , V = e aligned W V ;

[0046] [2] Calculating attention weights And calculate the weighted value A = Attention V;

[0047] [3] Use linear transformation and activation function to feed forward the weighted values ​​h ffn =ReLU(AW1+b1)W2+b2;

[0048] [4] Combine all transformed outputs to generate high-dimensional embeddings, e h =LayerNorm(e aligned +h ffn ).

[0049] Step S5: The large model uses the high-dimensional embedding as the input of the super network. Under the guidance of the high-dimensional embedding, the super network combines the image data of each drone for joint training to generate a small personalized image classification model for each drone.

[0050] The following steps are also included:

[0051] [1] On the server side, the hypernetwork first determines the number of layers and the number of neurons in each layer based on the target network. The number of participating nodes is denoted as n. The hypernetwork after t rounds of updates is denoted as h. t , the embedding vector of participating node i is denoted as e i , the embedding vector after large model optimization is recorded as e h , the super network according to e h The personalized model is denoted as w i ;

[0052] [2] According to the gradient descent method, the update of the hypernetwork can be expressed as α is the learning rate of the hypernetwork update, represents the gradient of the hypernetwork at node i;

[0053] [3] Due to Can write Then by the chain rule, It can be expressed as Therefore, the update of the hypernetwork can be written as

[0054] [4] During the hypernetwork update process, each node inputs the parameters into the local network for forward and backward propagation and updates the parameters. Calculate the change in the hypernetwork distribution parameters And upload to the server;

[0055] [5] The hypernetwork iteratively updates the network parameters based on the parameter changes uploaded by each node.

[0056] The super network training process is completed until convergence.

[0057] Step S6: In the inference stage, each drone uses a local personalized image classification model to classify the collected images and sends the classification results to the large model. In the decision stage, the large model performs final situation analysis and decision-making on the drone based on the original images and classification results.

[0058] The following steps are also included:

[0059] [1] Generate personalized model parameters θ for each client in the hypernetwork i After that, when the drone takes a new picture, it first uses the personalized model to get the classification result of the picture and sends it to the big model;

[0060] [2] The large model then combines the image with the classification results to analyze the situation of the drone in the current image and plan the next response measures.

[0061] In summary, the present invention proposes a method for generating a drone image classification model by integrating large and small models. A low-dimensional feature extraction module is designed in the large model to ensure the dimension and size consistency of the Embedding; a high-dimensional feature extraction module is designed to enable the Embedding to have stronger expression and discrimination capabilities through networks such as the attention mechanism; the high-dimensional Embedding is then input into the hypernetwork to generate a small image classification model; finally, through the classification results of the small model, the large model analyzes the operating status of the drone and generates countermeasures; the combination of large and small models can combine the strong text processing capabilities of the large model with the advantage of the small model in processing complex data.

[0062] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for generating a drone image classification model by integrating large and small models, characterized in that: The method comprises the following steps: S1. Design a large and small model architecture, which includes a server and multiple user nodes. The large model, super network and other modules are deployed on the server side, responsible for user question and answer, and the generation of small model for image classification. The small model for image classification is deployed on the drone node, responsible for real-time information collection, processing and training; S2. The drone is equipped with hardware such as a high-definition camera, GPS, and a microcontroller. An information acquisition module and a data preprocessing module are set in the microcontroller. The information acquisition module uses a high-definition camera and GPS to collect pictures and location information in real time. The data preprocessing module performs preprocessing operations such as data cleaning on the collected data. S3. Design low-dimensional feature extraction modules and high-dimensional feature extraction modules on the large model side, use the embedding network to extract low-dimensional features (Embedding) from the preprocessed image, and perform feature alignment after generating low-dimensional Embedding, including scale normalization, direction adjustment, etc., to ensure the consistency of their dimensions and sizes in subsequent processing; S4. Input the aligned low-dimensional embeddings into the high-dimensional feature extractor. Inside the high-dimensional feature extractor, the low-dimensional embeddings are transformed into high-dimensional embeddings through a series of transformations and reasoning operations such as self-attention mechanism and feedforward neural network. It has stronger expression and discrimination capabilities and can more comprehensively describe the complex content and structure in the image. S5. The large model uses the high-dimensional embedding as the input of the super network. Under the guidance of the high-dimensional embedding, the super network combines the image data of each drone for joint training to generate a small personalized image classification model for each drone. S6. In the inference stage, each drone uses a local personalized image classification model to classify the collected images and sends the classification results to the large model. In the decision-making stage, the large model performs final situation analysis and decision-making on the drone based on the original images and classification results.

2. The method for generating a drone image classification model by integrating large and small models according to claim 1 is characterized in that: The data preprocessing module in S2 performs preprocessing operations such as data cleaning on the collected operation data; First, the drone dataset is divided into training set, validation set and test set according to the preset ratio. Then, in the data preprocessing module, including image scaling, normalization, data enhancement, etc., the generalization ability of the model is improved, oversampling and undersampling are used to balance the imbalance of positive and negative sample data, and feature selection and polynomial feature generation are performed at the same time.

3. The method for generating a drone image classification model by integrating large and small models according to claim 1 is characterized in that: In S3, the low-dimensional feature extraction of the preprocessed image is performed using an embedding network, and the feature alignment process after generating the low-dimensional embedding includes the following steps: After preprocessing the data, the embedding network E in the large model is used to extract features to obtain a low-dimensional Embedding e=E(I), and then the mean μ is calculated through e e With standard deviation σ e , normalize e; After the normalization operation, the angle θ between the current feature vector and the target direction is adjusted according to the set target direction vector d to achieve the alignment of the low-dimensional Embedding and the high-dimensional Embedding.

4. The method for generating a drone image classification model by integrating large and small models according to claim 1, characterized in that: The low-dimensional embedding in S4 is converted into a high-dimensional embedding through a series of transformations and inference operations such as self-attention mechanism and feedforward neural network, including the following steps: First, use e aligned Calculate the attention query Q, K, V, calculate the attention weight Attention, and calculate the weighted value A; Then, A is feed-forwarded using a linear transformation and activation function h ffn =ReLU(AW1+b1)W2+b2; Finally, all transformed outputs are combined to generate a high-dimensional Embedding e h。 5. The method for generating a drone image classification model by integrating large and small models according to claim 1 is characterized in that: In S5, under the guidance of high-dimensional embedding, the super network is combined with the image data of each drone for joint training, and the generation of a personalized image classification model for each drone also includes the following steps: On the server side, the hypernetwork first determines the number of layers and the number of neurons in each layer to generate parameters based on the target network; then, the hypernetwork is updated using the gradient descent method; during the hypernetwork update process, each node inputs the parameters into the local network for forward and backward propagation and updates the parameters. Calculate the change in the hypernetwork distribution parameters And upload it to the server; finally, the super network iteratively updates the network parameters according to the parameter changes uploaded by each node until the super network training process is completed after convergence, and generates a personalized image classification small model for each drone.

6. The method for generating a drone image classification model by integrating large and small models according to claim 1, characterized in that: In the inference stage of S6, each drone uses a local personalized image classification model to classify the collected pictures and sends the classification results to the large model. In the decision stage, the large model performs a final situation analysis and decision on the drone based on the original pictures and the classification results. The following steps are involved: Generate personalized model parameters θ for each client in the hypernetwork i After that, when the drone takes a new picture, it first uses the personalized model to get the classification result of the picture and sends it to the big model; The large model then combines the image with the classification results to analyze the situation of the drone in the current image and plan the next response measures.

Images