Air-ground cooperative system and optimization method thereof
By building a collaborative system of cloud servers and edge drones, optimizing task allocation and data transmission, the problem of insufficient computing capabilities of drones is solved, the performance of edge models and overall system performance is improved, and the support for large models is achieved and updated.
Patent Information
- Application Number
- CN202411014371.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-26
- Publication Date
- 2024-12-03
AI Technical Summary
Existing UAVs (UAVs) as edge AI servers have insufficient computing power and cannot support large models that require billions of parameters, such as SoRA and Gemini, limit their practicality in 6G network applications and cannot model them. Updates, resulting in limited robustness and accuracy.
A space-ground collaboration system is proposed to optimize performance by building a collaborative system including cloud servers and edge drones, using task allocation, uplink or downlink bandwidth allocation, data transmission quantitative design and model update overhead of cloud servers and edge drones The objective function is to maximize the average accuracy average of drones and cloud servers and improve the performance of edge models.
Obtain perceived data through edge drones and perform feature extraction, transmit features and data residuals to cloud servers for inference, and send updated small models to drones, improving the performance of edge models on drones and improving the space and space The overall performance and computing power of the collaborative system.
Smart Images

Figure CN119071813A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of unmanned aerial vehicle technology, and in particular to an air-ground collaborative system and an optimization method thereof. Background Art
[0002] With the development of wireless communication technology, the sixth generation (6G) wireless network will achieve ubiquitous coverage by deploying air-ground collaborative facilities to realize a variety of applications such as surveillance, disaster response, and metaverse environment construction. The realization of the above applications relies on advanced technologies such as multimodal and generative artificial intelligence (AI), and requires support from large-scale inference models involving billions of parameters. These models not only require huge computing resources, but also need to maintain high inference accuracy and adaptability under various environmental conditions. This increasing demand for the deployment of computing facilities has promoted the research and development of deploying AI capabilities on edge computing devices such as unmanned aerial vehicles (UAVs).
[0003] Currently, although UAVs can be used as edge AI servers, due to the insufficient onboard computing power of UAVs, their applications as edge servers are usually limited to executing smaller inference models and cannot support large models that require billions of parameters, such as SoRA and Gemini, which severely limits their practicality in the expected 6G network applications. In addition, the computing power of edge UAVs is not sufficient to support model training, so edge models cannot be updated on UAVs, resulting in limited robustness and accuracy of edge AI servers.
[0004] Therefore, how to improve the computing power of UAV as an edge AI server is an urgent problem to be solved.
[0005] It should be noted that the above statements are only used to provide background technical information related to the present application and do not necessarily constitute prior art. Summary of the invention
[0006] In view of this, the purpose of this application is to propose an air-ground collaborative system and an optimization method thereof, which can solve the existing problems in a targeted manner.
[0007] Based on the above-mentioned purposes, in the first aspect, the present application proposes an optimization method for an air-ground collaborative system, including: constructing an air-ground collaborative system including a cloud server and an edge drone, wherein a cloud model is deployed in the cloud server, and an edge model is deployed in the edge drone; based on the average precision mean of the cloud model, the average precision mean of the edge model and the cloud server task allocation variable, constructing an objective function characterizing the performance of the air-ground collaborative system; and optimizing the objective function based on the task allocation of the cloud server and the edge drone, the uplink or downlink bandwidth allocation, the data transmission quantization design and the model update overhead.
[0008] In some embodiments, the objective function is optimized based on the task allocation, uplink or downlink bandwidth allocation, data transmission quantization design and model update overhead of the cloud server and edge drone, including: optimizing the objective function by maximizing the accuracy constraint, and the maximization accuracy constraint includes: the average accuracy mean function of the cloud model; the transmission ratio of residual data is less than or equal to the cloud server task allocation variable; the quantization bit number constraint of residual data transmission; the average accuracy mean function of the edge model; the update overhead constraint of the edge model; the transmission data size constraint of the feature stream; the transmission data size constraint of the data stream; the transmission data size constraint of the model stream; wherein the feature stream includes data transmitted by the drone to the cloud server, the data stream includes residual data with visual supplementary information, and the model stream includes model update data.
[0009] In some embodiments, the optimization of the objective function includes: constructing a data flow optimization function based on the residual data transmission ratio and the number of quantization bits for each pixel in each frame of data; constructing a feature flow and model flow joint optimization function based on cloud server task allocation variables, the bandwidth of drone uplink transmission, the bandwidth of drone downlink transmission, and model update overhead; respectively solving the optimal value of the data flow optimization function and the optimal value of the joint optimization function to obtain target parameter values corresponding to the data flow optimization function and the joint optimization function; optimizing the objective function according to the target parameter value to obtain the optimal solution of the objective function.
[0010] In some embodiments, a data flow optimization function is constructed based on the residual data transmission ratio and the number of quantization bits per pixel per frame data, and the method includes: obtaining an average precision mean function of the cloud model based on the residual data transmission ratio and the number of quantization bits per pixel per frame data; optimizing the residual data transmission ratio and the number of quantization bits per pixel per frame data to construct a data flow optimization function.
[0011] In some embodiments, the optimal value of the data flow optimization function and the optimal value of the joint optimization function are solved respectively to obtain the target parameter value, including: solving the data flow optimization function to obtain the optimal value of the average precision mean of the cloud model regarding the uplink transmission bandwidth of the UAV; judging whether the recall rate and the precision rate of the joint optimization function meet the preset conditions, and when the recall rate and the precision rate of the joint optimization function meet the preset conditions, solving the optimal value of the joint optimization function based on the optimal value of the average precision mean of the cloud model to obtain the target parameter values including cloud service task allocation variables, UAV uplink transmission bandwidth, UAV downlink transmission bandwidth, and model update overhead.
[0012] In some embodiments, when the recall rate and precision rate of the joint optimization function do not meet the preset conditions, the suboptimal value of the joint optimization function is solved based on the optimal value of the average precision mean of the cloud model, and the cloud service task allocation variables, drone uplink transmission bandwidth, drone downlink transmission bandwidth, and model update overhead corresponding to the suboptimal value are obtained as the target parameter values.
[0013] In some embodiments, the optimal value of the data flow optimization function and the optimal value of the joint optimization function are solved respectively to obtain the target parameter value, and it also includes: solving the data flow optimization function based on the value of the uplink transmission bandwidth of the drone to obtain the target parameter value including the transmission ratio of the residual data and the number of quantization bits of the residual data.
[0014] In some embodiments, determining whether the recall and precision of the joint optimization function meet preset conditions includes: respectively calculating the recall and precision of the cloud model at the kth intersection-and-union ratio, and the recall and precision of the edge model at the kth intersection-and-union ratio; based on the relationship between the difference between the recall rate of the cloud model and the recall rate of the edge model and the recall rate of the cloud model, and the relationship between the difference between the precision of the cloud model and the precision of the edge model and the precision of the cloud model, determining whether the recall and precision of the joint optimization function meet the preset conditions.
[0015] In the second aspect, an air-ground collaborative system is also provided, which includes: a cloud server and an edge drone; the edge drone has a data collector, the data collector is used to obtain perception data, an edge model is deployed in the edge drone, and the edge model is used to perform edge computing; the cloud server is used to perform calculations based on the perception data sent by the edge drone, the cloud model is deployed in the cloud server, and the cloud server uses the cloud model to update the edge model.
[0016] In some embodiments, the edge drone includes an edge reasoning model, a feature extraction model and a residual processing model; the edge drone receives a model stream including model update data from a cloud server through the edge reasoning model and the feature extraction model; the edge drone extracts features from the perception data through the feature extraction model as a feature stream, and transmits the feature stream to the cloud server; the edge drone transmits a data stream including residual data to the cloud server through the residual processing model, and the residual data includes visual supplementary information; the cloud server performs calculations based on the feature stream and the data stream.
[0017] In general, this application has at least the following beneficial effects:
[0018] Through an air-ground collaborative system including a cloud server and an edge drone, with the edge drone as the edge node and the cloud server as the central node, the edge drone obtains the perception data and performs feature extraction, and extracts the feature residuals of some perception data. The edge drone not only performs local reasoning calculations, but also transmits some extracted features and data residuals to the cloud server. The cloud server infers the received data based on the large model and sends the updated small model to the drone, thereby improving the performance of the edge model on the drone. Based on the task allocation of the cloud server and the edge drone, the uplink or downlink bandwidth allocation, the data transmission quantization design and the model update overhead, the average precision mean of the drone and the cloud server in the system is maximized, further improving the performance of the air-ground collaborative system.
[0019] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments disclosed in the present application and should not be regarded as limiting the scope of the present application. In addition, the same reference numerals are used to represent the same parts in all the drawings.
[0021] Figure 1 A schematic diagram of the system structure of an air-ground collaborative system provided by this embodiment is shown;
[0022] Figure 2 A schematic diagram of data transmission of an air-ground collaborative system provided by this embodiment is shown;
[0023] Figure 3 A flowchart showing a method for optimizing an air-ground collaborative system provided by this embodiment is shown;
[0024] Figure 4 A schematic diagram showing the distribution of perception data in this embodiment is shown;
[0025] Figure 5 Another schematic diagram showing the flow of the optimization method of the air-ground collaborative system in this embodiment;
[0026] Figure 6 Another flowchart of the optimization method of the air-ground collaborative system in this embodiment is shown. DETAILED DESCRIPTION
[0027] The following embodiments of the technical solution of the present application are described in detail in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present application, and are therefore only used as examples, and cannot be used to limit the scope of protection of the present application.
[0028] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by technicians in the technical field to which this application belongs; the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" in the specification and claims of this application and the above-mentioned figure descriptions and any variations thereof are intended to cover non-exclusive inclusions.
[0029] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0030] Existing unmanned aerial vehicles (UAVs) have insufficient onboard computing power. Their applications as edge servers are usually limited to executing smaller inference models and cannot support large models that require billions of parameters, such as SoRA and Gemini. This severely limits their practicality in expected 6G network applications. In addition, the computing power of edge UAVs is insufficient to support model training, so edge models cannot be updated on UAVs, resulting in limited robustness and accuracy of edge AI servers.
[0031] In response to the above problems, this embodiment provides an air-ground collaborative system and an optimization method thereof, wherein the air-ground collaborative system includes a cloud server and an edge drone, with the edge drone as the edge node and the cloud server as the central node. The edge drone obtains perception data and performs feature extraction, and extracts feature residuals of some perception data. The edge drone not only performs local inference calculations, but also transmits some extracted features and data residuals to the cloud server. The cloud server infers the received data based on the large model, and sends the updated small model to the drone, thereby improving the performance of the edge model on the drone.
[0032] At the same time, based on the above-mentioned air-ground collaborative system, this embodiment also proposes an optimization method, which maximizes the Mean Average Precision (MAP) of the UAV and the cloud server based on the task allocation of the cloud server and the edge UAV, the uplink or downlink bandwidth allocation, the data transmission quantization design, and the model update overhead, so as to further improve the performance of the air-ground collaborative system.
[0033] Figure 1 A schematic diagram of the system structure of an air-ground collaborative system provided in this embodiment is shown. Figure 1 ,An air-ground collaborative system includes: a cloud server and an edge drone.
[0034] The edge drone has a data collector, which is used to obtain perception data. The edge model is deployed in the edge drone, and the edge model is used for edge computing. The cloud server is used to perform calculations based on the perception data sent by the edge drone. The cloud model is deployed in the cloud server, and the cloud server uses the cloud model to update the edge model.
[0035] In one example, there may be one or more edge drones, and the number of edge drones may be set according to actual needs. The data collector is, for example, a camera, and correspondingly, the perception data is, for example, image data collected by the camera.
[0036] The edge model is used for edge computing. Taking the visual classification task as an example, the analysis and recognition of each frame of the video is called a task. The edge model can perform the visual classification task, and the cloud server can also perform the visual classification task.
[0037] After acquiring the perception data, the edge model sends the perception data to the cloud server. The cloud server performs calculations based on the perception data sent by the edge drone, and sends flexible cost model update data to the edge drone through the internally deployed cloud model to upgrade the edge model, thereby improving the performance of the edge model on the drone.
[0038] The cloud server is connected to the ground base station and can communicate with the edge drone through the Over-The-Air (OTA) network. Model streams, feature streams and data streams can be transmitted between the cloud server and the edge drone, where the model stream is the downlink transmission from the cloud serverless to the edge drone, and the feature stream and data stream are the uplink transmission from the edge drone to the cloud serverless.
[0039] Specifically, the edge drone of this embodiment includes an edge reasoning model, a feature extraction model, and a residual processing model; the edge drone receives a model stream including model update data from a cloud server through the edge reasoning model and the feature extraction model. The edge drone extracts features from the perception data through the feature extraction model as a feature stream, and transmits the feature stream to the cloud server; the edge drone transmits a data stream including residual data to the cloud server through the residual processing model, and the residual data includes visual supplementary information; the cloud server performs calculations based on the feature stream and the data stream.
[0040] The edge inference model can perform edge computing to enable the deployment of convolutional neural networks in the air, improve edge computing capabilities, and distribute tasks to aerial devices to accommodate technologies such as multimodal and generative artificial intelligence (AI).
[0041] For tasks and data in different fields, the feature extraction model and edge inference model of the edge drone can be upgraded by receiving model update data with flexible overhead from the cloud server. These model update data are called model streams.
[0042] The edge drone extracts perception data using an onboard feature extraction model and then transmits the extracted visual data features to the cloud server via OTA, which is called feature streaming.
[0043] In order to further improve the inference performance of the cloud server and maintain high inference accuracy and adaptability, this embodiment uses idle OTA transmission resources to transmit more detailed visual supplementary information than the feature stream to the cloud server, namely residual data. These data can be transmitted at an adjustable resolution and are called data streams.
[0044] The cloud server performs calculations based on the feature flow and data flow, generates new model update data, updates the model flow, and realizes a closed loop of system data.
[0045] Figure 2 A schematic diagram of data transmission of an air-ground collaborative system provided by this embodiment is shown, referring to Figure 2 , the edge drone obtains the perception data, extracts the features of the perception data, and forms a feature stream. At the same time, it transmits more detailed visual supplementary information than the feature stream to the cloud server as residual data to form a data stream. The edge drone transmits the above feature stream and data stream to the cloud server. The cloud server uses the inference big model to calculate according to the feature stream and data stream, and generates model update data to form a model stream. The feature extraction model and edge inference model of the edge drone then use the model stream to update the model. In this way, the edge drone can independently perform edge computing to generate feature streams and data streams to support various edge services, such as disaster response and geographic identification. At the same time, the cloud server can provide model support for the edge drone through the model stream to improve the computing power of the edge drone.
[0046] Based on the air-ground coordination system provided in the above embodiment, this embodiment further provides an optimization method for the air-ground coordination system. Figure 3 A flowchart showing the steps of an optimization method for an air-ground collaborative system provided in this embodiment is shown. Figure 3 , an optimization method for an air-ground collaborative system comprises the following steps:
[0047] S301. Build an air-ground collaborative system including a cloud server and edge drones.
[0048] The structure of the air-ground collaborative system of this embodiment and its construction method have been described in the above embodiments. To avoid repetition, they are not described here again. Figure 1 and 2The air-ground collaborative system shown in the figure, in which the cloud model is deployed in the cloud server and the edge model is deployed in the edge drone.
[0049] S302: Based on the average precision of the cloud model, the average precision of the edge model and the cloud server task allocation variables, an objective function characterizing the performance of the air-ground collaborative system is constructed.
[0050] In this embodiment, the performance of the air-ground collaborative system can be represented by the mean average precision mAP of the system, that is, the mean average precision mAP of the system is obtained by the mean average precision mAP of the cloud model. L , the average precision mAP of the edge model S and the cloud server task allocation variable β, the experimental fitting function of mAP is represented by function f(·), so the objective function characterizing the performance of the air-ground collaborative system can be expressed as:
[0051] mAP = f(mAP L , mAP S , β)
[0052] The following is an explanation of the parameters involved in this embodiment:
[0053] Figure 4 FIG. 4 shows a schematic diagram of the distribution of the sensing data in this embodiment. Figure 4 As shown, it is assumed that the drone camera in this embodiment captures images at a frequency of N frames per second, where each frame contains x pixels and each pixel is quantized to b bits. Therefore, the visual data generation rate of the drone is D = N·x·b. A portion of the frames with a ratio of β are uploaded to the cloud server via OTA transmission for visual target classification, while the remaining (1-β) proportion of frames are processed on the drone using an edge model. In order to improve the classification accuracy on the cloud server, the residual data of a small portion of the frames with a ratio of ρ can be transmitted from the drone to the cloud server.
[0054] That is to say, this embodiment uses β in the perception data as the cloud server task allocation variable, (1-β) as the edge computing data of the drone, and a part of the proportion ρ in β as the residual data.
[0055] In this way, Ψ represents the frame set analyzed on the cloud server, and |Ψ|=βN. The features of the frame set Ψ are extracted on the drone for subsequent processing on the cloud server. The average size of the features extracted for the next frame, the transmission data rate of the feature stream is given by the following formula:
[0056]
[0057] Let Φ represent the set of frames whose residual data are sent to the cloud server, and |Φ| = ρN. Since the residual data needs to be combined with the features extracted on the cloud server for image reconstruction, the set Φ only contains the frames analyzed on the cloud server, that is, In addition, due to the limited OTA transmission bandwidth, it is necessary to properly quantize the residual data. is the quantization parameter of the residual data, where Represents the number of quantization bits for each pixel in the i-th frame, which is selected from a set of discrete values and is denoted by Ω. The transmission data rate of the data stream can be expressed as:
[0058]
[0059] The sum of the transmission data rates of the feature stream and the data stream should not exceed the upload capacity limit of the drone, that is:
[0060] R F +R D ≤B u ·S u
[0061] Among them, B u is the bandwidth of the UAV uplink transmission, S u is the spectrum efficiency of transmission on the UAV. The spectrum efficiency can be considered as a usable value through appropriate channel measurement techniques without being affected by the wireless propagation environment.
[0062] The cloud server accumulates a large amount of feature and residual data from the edge drones, and is able to update the edge model to improve the inference accuracy of the edge drones. The model update information is transmitted to the drone with an average overhead of M bits per second, which represents the data rate of the model stream. It is worth noting that the data rate of the model stream cannot exceed the download capacity of the drone, that is:
[0063] M≤B d ·S d
[0064] Among them, B d is the bandwidth of the UAV downlink transmission, S d is the spectrum efficiency of the drone downlink transmission. At the same time, assume that the total bandwidth allocated to the drone-base station communication is B and satisfies:
[0065] B u +B d ≤B
[0066] In this embodiment, the spectrum efficiency S of uplink and downlink transmission is u and S d is regarded as a constant with an arbitrary value, since S u and S dThe size of S may be affected by the drone trajectory, transmit beamforming, and interference management. Therefore, this embodiment does not specifically limit S u and S d The size can be set according to actual conditions.
[0067] Assuming that the cloud model on the cloud server is well trained and has stable performance, the average mean accuracy (mAP) of the cloud model is L Determined by the quality of the features and residual data transmitted by the drone, a fixed-cost feature extraction method is applied to each frame in the set Ψ. mAP of the cloud model L is a function of the residual data transmission ratio ρ and the corresponding quantization bit, mAP L The experimental fitting function of the cloud model is represented by the function g(·), then the average precision mAP of the cloud model is L It is expressed as:
[0068]
[0069] For edge drones, which can obtain lossless data of all frames, the inference accuracy of the edge model affects the mAP performance, and the inference accuracy of the edge model is determined by the model update provided by the cloud server. S The experimental fitting function of is represented by the function h(·), then the average precision mAP of the edge model is S It is expressed as:
[0070] mAP S =h(M),M min ≤M≤M max
[0071] Where M is the model update cost, M min and M max are the minimum and maximum model update costs, respectively.
[0072] Get the mean average precision mAP of the cloud model L , the average precision mAP of the edge model S After assigning the variable β to the cloud server task, we can get the objective function mAP = f(mAP L , mAP S , β).
[0073] S303, optimize the objective function based on the task allocation of cloud servers and edge drones, uplink or downlink bandwidth allocation, data transmission quantization design, and model update overhead.
[0074] In order to maximize the mAP of the air-ground collaborative system of this embodiment, this embodiment combines the task allocation of the cloud server and the edge drone, the uplink or downlink bandwidth allocation, the data transmission quantization design, and the model update overhead to optimize the above objective function.
[0075] Specifically, the objective function is optimized by maximizing the accuracy constraint condition, and the maximizing accuracy constraint condition includes:
[0076] The average precision mean function of the cloud model is
[0077] The transmission ratio of residual data is less than or equal to the cloud server task allocation variable, that is, 0≤ρ≤β≤1;
[0078] The quantization bit number constraint for residual data transmission is
[0079] The average precision function of the edge model, i.e. mAP S =h(M);
[0080] The update cost constraint of the edge model, i.e., M min ≤M≤M max ;
[0081] The transmission data size constraint of the feature flow, i.e. R F +R D ≤B u ·S u ;
[0082] The transmission data size constraint of the data stream, that is, M≤B d ·S d ;
[0083] The transmission data size constraint of the model flow, that is, B u +B d ≤B;
[0084] Among them, the feature stream includes the data transmitted by the drone to the cloud server, the data stream includes the residual data with visual supplementary information, and the model stream includes the model update data, which have been described above and will not be repeated here.
[0085] That is to say, the optimization problem of the objective function of this embodiment can be modeled as:
[0086]
[0087] 0≤ρ≤β≤1,
[0088]
[0089] mAP S =h(M),
[0090] M min ≤M≤M max ,
[0091] RF +R D ≤B u ·S u ,
[0092] M≤B d ·S d ,
[0093] B u +B d ≤B.
[0094] By solving the optimal solution of the above optimized objective function, we can obtain the cloud service task allocation variable β, the transmission ratio ρ of the residual data, the quantization bit number b of the residual data, and the bandwidth B of the UAV uplink transmission when the total uplink and downlink bandwidth and spectrum efficiency of the wireless network are given. u , bandwidth of drone downlink transmission B d , the optimal solution of the model update cost M can be obtained to maximize the mAP of the air-ground collaborative system.
[0095] Furthermore, in order to simplify the solution of the above optimization problem, this embodiment decomposes the above optimization problem into two sub-problems: a data flow design sub-problem and a feature flow / model flow design sub-problem, and analyzes these two sub-problems in turn.
[0096] Specifically, Figure 5 Another flowchart of the optimization method of the air-ground collaborative system in this embodiment is shown.
[0097] refer to Figure 5 , the optimization objective function of this embodiment includes the following steps:
[0098] S501: construct a data flow optimization function based on the residual data transmission ratio and the number of quantization bits per pixel per frame of data.
[0099] Specifically, based on the residual data transmission ratio and the number of quantization bits per pixel per frame data, a data flow optimization function is constructed, and the method includes: based on the residual data transmission ratio and the number of quantization bits per pixel per frame data, obtaining the average precision mean function of the cloud model; optimizing the residual data transmission ratio and the number of quantization bits per pixel per frame data, and constructing a data flow optimization function.
[0100] In this embodiment, the residual data transmission ratio is ρ, and the number of quantization bits per pixel per frame data is b. Based on the residual data transmission ratio and the number of quantization bits per pixel per frame data, the average precision mean function of the cloud model is obtained, that is,
[0101] The residual data transmission ratio and the number of quantization bits per pixel of each frame data are optimized to construct a data flow optimization function, that is, the mAP of the cloud model is maximized by optimizing the ratio of frames with residual data transmission and their corresponding number of quantization bits, which can be expressed as:
[0102] max ρ,b mAP = mAP L ,
[0103]
[0104] 0≤ρ≤β,
[0105]
[0106] R F +R D ≤B u ·S u .
[0107] Among them, since the number of quantization bits of each frame may be different, the mAP of different frames may be different. represents the mAP of the cloud model when analyzing the i-th frame. It can be understood intuitively that As the number of quantization bits of its residual data Monotonically increasing. Without considering the discrete quantization bit constraint Maximize mAP in the case of L The solution satisfies
[0108] when is the number of quantization bits concave function, treating it as a continuous variable, where In the case where there is no residual data transmission, the residual data of all frames in Ψ should be sent to the cloud server with the same number of quantization bits, that is, Therefore, the above data flow design sub-problem can be solved as follows:
[0109] In the variable R F , B u and S u Given, R F +R D ≤B u ·S u Can be converted into When satisfied is the number of quantization bits A concave function of When, first set ρ = β and the optimized if The value of does not satisfy the constraint From and Select the value of the element in b, where and The two closest value, and satisfies and have The ratio is quantified as bits of residual data, and The ratio is quantified as The residual data of the bit is 1, where “·” is the function to obtain the nearest integer.
[0110] When the above assumptions are not met, this embodiment also proposes a heuristic algorithm, which includes calculating the maximum data rate of residual data transmission, denoted as B u ·S u -R F The concept of mAP incremental efficiency per frame is introduced, which represents the increase in mAP with the unit increment of the number of data quantization bits. This strategy preferentially allocates the remaining communication resources to the frames with the highest mAP incremental efficiency. This iterative process continues until all communication resources are effectively allocated.
[0111] S502: Based on the cloud server task allocation variables, the bandwidth of the drone uplink transmission, the bandwidth of the drone downlink transmission, and the model update overhead, a joint optimization function of the feature flow and the model flow is constructed.
[0112] The joint optimization function of feature flow and model flow can be expressed as:
[0113]
[0114] st0≤ρ≤β≤1,
[0115] mAP S =h(M),
[0116] M min ≤M≤M max ,
[0117] R F +R D ≤B u ·S u ,
[0118] M≤B d ·S d ,
[0119] B u +B d ≤B.
[0120] S503 , respectively solving the optimal value of the data flow optimization function and the optimal value of the joint optimization function to obtain target parameter values corresponding to the data flow optimization function and the joint optimization function.
[0121] Step S503 includes: solving the data flow optimization function to obtain the optimal value of the average precision mean of the cloud model regarding the uplink transmission bandwidth of the UAV; judging whether the recall rate and precision rate of the joint optimization function meet the preset conditions, and when the recall rate and precision rate of the joint optimization function meet the preset conditions, solving the optimal value of the joint optimization function based on the optimal value of the average precision mean of the cloud model to obtain target parameter values including cloud service task allocation variables, UAV uplink transmission bandwidth, UAV downlink transmission bandwidth, and model update overhead.
[0122] Solve the data flow optimization function to obtain the optimal value of the average accuracy of the cloud model for the uplink transmission bandwidth of the drone, that is, by solving the max ρ,b mAP = mAP L , get the bandwidth B of the UAV uplink transmission u mAP L When the recall rate and precision rate of the joint optimization function meet the preset conditions, the optimal value mAP based on the average precision of the cloud model L Solving the Joint Optimization Function The optimal value of the cloud service task allocation variable β and the UAV uplink transmission bandwidth B u 、UAV downlink transmission bandwidth B d , the target parameter value of the model update cost M.
[0123] Among them, judging whether the recall rate and precision rate of the joint optimization function meet the preset conditions, including: respectively calculating the recall rate and precision rate of the cloud model under the kth intersection and union ratio, and the recall rate and precision rate of the edge model under the kth intersection and union ratio; judging whether the recall rate and precision rate of the joint optimization function meet the preset conditions based on the relationship between the difference between the recall rate of the cloud model and the recall rate of the edge model and the recall rate of the cloud model, and the relationship between the difference between the precision rate of the cloud model and the precision rate of the edge model and the precision rate of the cloud model.
[0124] The joint mAP of the cloud model and the edge model is a function of the recall-precision pair of the cloud model and the edge model, which can be expressed as:
[0125]
[0126] in, and are the recall and precision of the cloud model at the kth intersection over union (IoU), respectively, and and are respectively the recall and precision of the edge model at the kth IoU. In addition, the joint mAP satisfies:
[0127]
[0128] The above formula gives mAP as mAP L , mAP S The goal of this invention is to establish mAP and mAP under specific conditions. L , mAP S and β. In fact, when the constraint When, the cloud model Much larger than marginal models Cloud Model Much larger than marginal models The joint mAP of the cloud model and edge model is approximately:
[0129]
[0130] The above formula focuses on the scenario where the mAP performance of the cloud model and the edge model is of the same order of magnitude, providing a closed-form expression for mAP. When it is not strictly satisfied, the above formula can still be regarded as the lower bound of mAP to solve the feature / model flow design sub-optimization problem. About mAP L and mAP S The convexity of can be obtained by calculating its Hessian matrix, that is:
[0131]
[0132] As can be seen from the above formula, the first-order and second-order principal minors of the Hessian matrix are non-positive, so It’s about mAP L and mAP S is a concave function. Since mAP L It's about R D is a concave function, and R F The value of does not affect mAP L , so mAP L It can be regarded as about R F +R D In addition, due to the uplink transmission data size R F +R D is the uplink transmission bandwidth B u A linear function of , so mAP L Also about B u is a concave function. When satisfied, mAP L and mAP S is concave, and h(·) is fitted as a concave function. Therefore, this subproblem is about β, B d , B u , is a concave function of M and can be solved using convex optimization methods.
[0133] In this embodiment, when the recall rate and precision rate of the joint optimization function do not meet the preset conditions, that is, when they do not meet the preset conditions, When the optimal value of the average precision mean of the cloud model is used to solve the suboptimal value of the joint optimization function, the cloud service task allocation variable β and the UAV uplink transmission bandwidth B corresponding to the suboptimal value are obtained. u 、UAV downlink transmission bandwidth B d , the model update cost M is used as the target parameter value.
[0134] That is, the recall and precision of the joint optimization function do not meet When Solve the approximate function f(·) to obtain the lower bound solution.
[0135] In the optimization method of the air-ground collaborative system of this embodiment, the optimal value of the data flow optimization function and the optimal value of the joint optimization function are solved respectively to obtain the target parameter value, and it also includes: solving the data flow optimization function based on the value of the uplink transmission bandwidth of the unmanned aerial vehicle to obtain the target parameter value including the transmission ratio ρ of the residual data and the quantization bit number b of the residual data.
[0136] S504: Optimize the objective function according to the target parameter value to obtain the optimal solution of the objective function.
[0137] The cloud service task allocation variable β and the UAV uplink transmission bandwidth B can be obtained by solving the above step S503. u 、UAV downlink transmission bandwidth B d , model update overhead M, residual data transmission ratio ρ and residual data quantization bit number b, then β, B u , B d , M, ρ, and b as the target parameter values, we can solve Obtain the optimal solution for maximizing the average precision of the cloud-edge collaborative system.
[0138] Figure 6 Another flow chart of the optimization method of the air-ground collaborative system in this embodiment is shown. Figure 6 The present invention first calculates the total bandwidth B of the drone-base station communication and the spectrum efficiency S of the drone downlink transmission by inputting variables. d, a discrete value Ω representing the number of quantization bits per pixel, and the minimum and maximum model update costs M min and M max As well as functions g(·) and h(·), by solving the data flow optimization function max ρ,b mAP = mAP L , get the bandwidth B of the UAV uplink transmission u mAP L The optimal value of Whether the recall rate and precision rate of When established, you can use The optimal solution is obtained, and the cloud service task allocation variable β and the UAV uplink transmission bandwidth B corresponding to the optimal solution of the joint optimization function are u 、UAV downlink transmission bandwidth B d , the model update cost M is used as the target parameter value. Under the condition When it is not true, the actual mAP value is not less than Then you can use The suboptimal solution is obtained. The cloud service task allocation variable β and the UAV uplink transmission bandwidth B corresponding to the suboptimal solution of the joint optimization function are u 、UAV downlink transmission bandwidth B d , the model update cost M is used as the target parameter value.
[0139] Then the UAV uplink transmission bandwidth B u The solution is re-substituted into the data flow optimization function max ρ,b mAP = mAP L , and obtain the target parameter values of the transmission ratio ρ of the residual data and the quantization bit number b of the residual data. The final target parameter values include the cloud service task allocation variable β, the UAV uplink transmission bandwidth B u 、UAV downlink transmission bandwidth B d , model update overhead M, residual data transmission ratio ρ and residual data quantization bit number b, we can solve Obtain the optimal solution for maximizing the average precision of the cloud-edge collaborative system.
[0140] The above is an optimization method for an air-ground collaborative system provided in this embodiment. Based on the data transmission type and mode of the air-ground collaborative system, the average precision mean function expression of the air-ground collaborative system is obtained, and the average mAP of the drone and the cloud server is maximized through task allocation, transmission resource allocation, transmission data quantization design and edge model update overhead. It can maximize the joint reasoning capability of the cloud server model and the drone edge model when the total uplink and downlink bandwidth and spectrum efficiency of the wireless network are given, thereby improving the overall mAP of the system.
[0141] The subject for executing the optimization method of the air-ground collaborative system in the embodiment of the present application may be an electronic device capable of executing the battery visual inspection confirmation method, and the electronic device may include but is not limited to a terminal or a server.
[0142] It should be noted that:
[0143] In the above text, the term "comprises", "includes" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be noted that the scope of the method and device in the embodiment of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.
[0144] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.
[0145] The embodiments of the present application are described above in conjunction with the accompanying drawings, which are only specific implementation modes of the present application. However, the present application is not limited to the above-mentioned specific implementation modes. The above-mentioned specific implementation modes are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.
Claims
1. An optimization method for an air-ground collaborative system, characterized in that: include: Constructing an air-ground collaborative system including a cloud server and an edge drone, wherein a cloud model is deployed in the cloud server and an edge model is deployed in the edge drone; Based on the average precision mean of the cloud model, the average precision mean of the edge model and the cloud server task allocation variable, construct an objective function characterizing the performance of the air-ground collaborative system; The objective function is optimized based on the task allocation of the cloud server and the edge drone, the uplink or downlink bandwidth allocation, the data transmission quantization design, and the model update overhead.
2. The optimization method of the air-ground collaborative system according to claim 1, characterized in that: The optimizing the objective function based on the task allocation of the cloud server and the edge drone, the uplink or downlink bandwidth allocation, the data transmission quantization design, and the model update overhead includes: optimizing the objective function by maximizing the accuracy constraint condition, and the maximizing accuracy constraint condition includes: Average precision mean function of cloud model; The transmission ratio of residual data is less than or equal to the cloud server task allocation variable; Constraints on the number of quantization bits for residual data transmission; Average precision mean function of marginal models; Update cost constraints for edge models; The transmission data size constraints of feature streams; The transmission data size constraints of the data stream; The transmission data size constraints of the model flow; The feature stream includes data transmitted from the drone to the cloud server, the data stream includes residual data with visual supplementary information, and the model stream includes model update data.
3. The optimization method of the air-ground collaborative system according to claim 1 or 2, characterized in that: The optimizing the objective function comprises: Construct a data flow optimization function based on the residual data transmission ratio and the number of quantization bits per pixel in each frame of data; Based on the cloud server task allocation variables, the bandwidth of the drone uplink transmission, the bandwidth of the drone downlink transmission, and the model update overhead, a joint optimization function of the feature flow and the model flow is constructed; Solving the optimal value of the data flow optimization function and the optimal value of the joint optimization function respectively to obtain target parameter values corresponding to the data flow optimization function and the joint optimization function; The objective function is optimized according to the objective parameter value to obtain an optimal solution of the objective function.
4. The optimization method of the air-ground collaborative system according to claim 3 is characterized in that: Based on the residual data transmission ratio and the number of quantization bits per pixel of each frame data, a data flow optimization function is constructed, and the method includes: Based on the residual data transmission ratio and the number of quantization bits per pixel in each frame of data, the average precision mean function of the cloud model is obtained; The residual data transmission ratio and the number of quantization bits per pixel per frame of data are optimized to construct a data flow optimization function.
5. The optimization method of the air-ground collaborative system according to claim 3, characterized in that: Solving the optimal value of the data flow optimization function and the optimal value of the joint optimization function respectively to obtain the target parameter value includes: Solving the data flow optimization function to obtain the optimal value of the average precision of the cloud model with respect to the uplink transmission bandwidth of the UAV; Determine whether the recall rate and precision rate of the joint optimization function meet the preset conditions. If the recall rate and precision rate of the joint optimization function meet the preset conditions, solve the optimal value of the joint optimization function based on the optimal value of the average precision mean of the cloud model to obtain target parameter values including cloud service task allocation variables, drone uplink transmission bandwidth, drone downlink transmission bandwidth, and model update overhead.
6. The optimization method of the air-ground collaborative system according to claim 5, characterized in that: When the recall rate and precision rate of the joint optimization function do not meet the preset conditions, the suboptimal value of the joint optimization function is solved based on the optimal value of the average precision mean of the cloud model, and the cloud service task allocation variables, drone uplink transmission bandwidth, drone downlink transmission bandwidth, and model update overhead corresponding to the suboptimal value are obtained as the target parameter values.
7. The optimization method of the air-ground collaborative system according to any one of claims 5 or 6, characterized in that: Solving the optimal value of the data flow optimization function and the optimal value of the joint optimization function respectively to obtain the target parameter value also includes: The data flow optimization function is solved based on the value of the UAV uplink transmission bandwidth to obtain target parameter values including the transmission ratio of the residual data and the number of quantization bits of the residual data.
8. The optimization method of the air-ground collaborative system according to claim 5, characterized in that: Determining whether the recall rate and precision rate of the joint optimization function meet preset conditions includes: Respectively calculating the recall rate and precision rate of the cloud model at the kth intersection-and-union ratio, and the recall rate and precision rate of the edge model at the kth intersection-and-union ratio; Based on the relationship between the difference between the recall rate of the cloud model and the recall rate of the edge model and the recall rate of the cloud model, as well as the relationship between the difference between the precision rate of the cloud model and the precision rate of the edge model and the precision rate of the cloud model, it is determined whether the recall rate and precision rate of the joint optimization function meet the preset conditions.
9. An air-ground collaborative system, characterized in that: The system includes: a cloud server and an edge drone; The edge drone has a data collector, which is used to obtain perception data. An edge model is deployed in the edge drone, and the edge model is used to perform edge computing; The cloud server is used to perform calculations based on the perception data sent by the edge drone. A cloud model is deployed in the cloud server, and the cloud server uses the cloud model to update the edge model.
10. The air-ground collaborative system according to claim 9, characterized in that: The edge drone includes an edge reasoning model, a feature extraction model, and a residual processing model; The edge drone receives a model stream including model update data from a cloud server through the edge inference model and the feature extraction model; The edge drone extracts features from the perception data through the feature extraction model as a feature stream, and transmits the feature stream to the cloud server; The edge drone transmits a data stream including residual data to a cloud server through the residual processing model, wherein the residual data includes visual supplementary information; The cloud server performs calculations based on the feature stream and the data stream.
