Deployment method and device of multi-modal remote sensing image interpretation model, medium and equipment

Abstract

The invention discloses a deployment method and device of a multi-modal remote sensing image interpretation model, a medium and equipment, and relates to the technical field of deep learning. The method comprises the following steps: acquiring a reference multi-modal remote sensing image interpretation model; the reference multi-modal remote sensing image interpretation model comprises a feature extraction model and a feature interpretation model; matrix multiplication in the feature extraction model is replaced with block assembly line matrix multiplication, and an optimized feature extraction model is obtained; the block assembly line matrix multiplication comprises the steps of respectively partitioning two matrixes to be multiplied, and sequentially processing different block multiplications according to a block assembly line mode; and deploying the optimized feature extraction model on a field programmable gate array (FPGA) in a heterogeneous computing platform of the unmanned aerial vehicle, performing model quantization processing on the feature interpretation model, and deploying the quantized feature interpretation model on a graphics processing unit (GPU) in the heterogeneous computing platform of the unmanned aerial vehicle. The multi-modal remote sensing data interpretation model deployed through the deployment method is high in interpretation efficiency.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a deployment method, device, medium and equipment for a multimodal remote sensing image interpretation model of heterogeneous computing hardware of an unmanned platform. Background Art

[0002] UAV Low Altitude Remote Sensing System (UAVLARS) is a new type of remote sensing platform with high mobility and wide application fields. It uses common digital remote sensing aerial survey equipment mounted on the UAV platform to shoot and record the earth's surface images along the flight trajectory, and uses the remote sensing data processing system to process the data and apply it to various industries.

[0003] At present, most drones adopt the solution of collecting images and then transmitting them back to the terminal for processing. The delay of wireless transmission seriously affects the real-time analysis results. If the drone itself has the ability of intelligent image recognition, it can flexibly adjust the flight plan according to the image processing results, and the response time will be greatly reduced, which improves the intelligence level of drones and has good application prospects in the field of emergency rescue. In recent years, with the development of integrated circuit technology and artificial intelligence technology, the computing resources of embedded systems have been greatly improved. Field-Programmable Gate Array (FPGA)-Graphics Processing Unit (GPU) heterogeneous computing hardware is deployed on drones, and image interpretation is performed through the multimodal remote sensing data interpretation model deployed on FGPA-GPU heterogeneous computing hardware.

[0004] However, the interpretation efficiency of the multimodal remote sensing data interpretation model deployed on FGPA-GPU heterogeneous computing hardware in the related art is low. Summary of the invention

[0005] Based on this, it is necessary to provide a deployment method, device, medium and equipment for a multimodal remote sensing image interpretation model to address the above-mentioned technical problems. The multimodal remote sensing data interpretation model deployed by this deployment method has a high interpretation efficiency.

[0006] The present invention adopts the following technical solutions:

[0007] The present invention provides a deployment method of a multimodal remote sensing image interpretation model, comprising:

[0008] Acquire a benchmark multimodal remote sensing image interpretation model; the benchmark multimodal remote sensing image interpretation model includes a feature extraction model and a feature interpretation model;

[0009] The matrix multiplication in the feature extraction model is replaced by a block pipeline matrix multiplication to obtain an optimized feature extraction model; the block pipeline matrix multiplication includes dividing two matrices to be multiplied into blocks respectively, and processing different block multiplications in sequence according to the block pipeline method;

[0010] The optimized feature extraction model is deployed on the field programmable gate array FPGA in the heterogeneous computing platform of the drone, and the feature interpretation model is quantized and deployed on the graphics processing unit GPU in the heterogeneous computing platform of the drone.

[0011] Preferably, the feature extraction model includes an average value calculation module, a covariance matrix calculation module, an eigenvalue decomposition module, a sorting module and a projection module; the specific implementation of the feature extraction model includes:

[0012] For any multimodal remote sensing image, the multimodal remote sensing image is vectorized through the average value calculation module to obtain the image matrix Multimodal remote sensing images have p bands and a spatial dimension of m×n. is an N-dimensional column vector; calculate the average value of each row of data in the image matrix X to obtain the mean matrix x ij is the data of the i-th row and j-th column in the matrix, is the average value of all data in the i-th row;

[0013] Through the covariance matrix calculation module, each data in the image matrix X is subtracted from the average value of the same row of data to obtain the center translation matrix Y: And calculate the covariance matrix of the center translation matrix Y:

[0014] The eigenvalue decomposition module is used to obtain the eigenvalues ​​of the covariance matrix and the corresponding mutually orthogonal unit eigenvectors, and the sorting module is used to form an eigenvector matrix using the M eigenvectors corresponding to the largest M eigenvalues.

[0015] The projection matrix is ​​determined by the projection module based on the eigenvector matrix V and the center translation matrix Y.

[0016]

[0017] Preferably, the matrix multiplication in the feature extraction model is replaced with a block pipeline matrix multiplication, including:

[0018] The center translation matrix Y of size p×N is divided into m blocks in sequence, each block has b rows, m and b satisfy p=m×b, and the transposed matrix of the center translation matrix Y is divided into m blocks, each block has b columns;

[0019] Process different block multiplications in sequence according to the block pipeline method to obtain the covariance matrix;

[0020] Update the feature extraction model based on how the covariance matrix is ​​calculated.

[0021] Preferably, the m blocks include M1, M2, and M3; and different block multiplications are processed in sequence in a block pipeline manner to obtain a covariance matrix, including:

[0022] The data in M1, M2, and M3 flow into the random access memory RAM1 in blocks. After the data in the blocks are stored in RAM1, M is calculated and obtained respectively. 11 、M 22 、M 33 ;

[0023] Control the data in RAM1 to remain unchanged, store the data of M1 and M2 into the random access memory RAM2 in turn, and calculate M 13 、M 31 、M 23 and M 32 ;

[0024] Store the data of M2 in RAM1 and the data of M1 in RAM2, and calculate M 12 and M 21 .

[0025] Preferably, the feature interpretation model is subjected to model quantization processing, including:

[0026] Determine the quantization bit width of weights and activation values ​​of each layer in the feature interpretation model based on reinforcement learning and hardware accelerators;

[0027] The feature interpretation model is updated by the quantization bit width of the weights and activation values ​​of each layer to obtain a quantized feature interpretation model.

[0028] Preferably, the method further comprises:

[0029] Before model deployment, the feature extraction model and feature interpretation model in the benchmark multimodal remote sensing image interpretation model are respectively executed on different computing units in the heterogeneous computing hardware of the UAV to obtain the running time of the feature extraction model and the feature interpretation model on different computing units; the computing units include FPGA and GPU;

[0030] According to the actual communication rate between the computing units and the running time of each module executed on different computing units, the optimal computing units for executing the feature extraction model and the feature interpretation model are determined respectively.

[0031] The present invention provides a deployment device for a multimodal remote sensing image interpretation model, comprising:

[0032] An acquisition module is used to acquire a benchmark multimodal remote sensing image interpretation model; the benchmark multimodal remote sensing image interpretation model includes a feature extraction model and a feature interpretation model;

[0033] An optimization module, used for replacing matrix multiplication in the feature extraction model with block pipeline matrix multiplication to obtain an optimized feature extraction model; the block pipeline matrix multiplication includes dividing two matrices to be multiplied into blocks respectively, and processing different block multiplications in sequence according to the block pipeline method;

[0034] The deployment module is used to deploy the optimized feature extraction model on the field programmable gate array FPGA in the heterogeneous computing platform of the drone, and to quantize the feature interpretation model, and then deploy the quantized feature interpretation model on the graphics processing unit GPU in the heterogeneous computing platform of the drone.

[0035] The present invention provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the deployment method of the multimodal remote sensing image interpretation model is implemented.

[0036] The present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the deployment method of the multimodal remote sensing image interpretation model when executing the program.

[0037] At least one of the above-mentioned technical solutions adopted in the present invention can achieve the following beneficial effects: This application adopts a heterogeneous computing mode deployment solution to replace the matrix multiplication in the feature extraction model in the multimodal remote sensing data interpretation model with a block pipeline matrix multiplication, that is, the two matrices to be multiplied are divided into blocks, and different block multiplications are processed in sequence according to the block pipeline method. In this way, by designing different calculation stages into a pipeline structure, efficient parallel processing can be achieved, which can improve the computational efficiency of the feature extraction model, thereby improving the interpretation efficiency of the multimodal remote sensing data interpretation model.

[0038] Furthermore, the feature interpretation model in the multimodal remote sensing data interpretation model is quantized to reduce the model size and improve the model reasoning speed, thereby further improving the interpretation efficiency of the multimodal remote sensing data interpretation model. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0040] Figure 1A schematic diagram of a deployment method of a multimodal remote sensing image interpretation model provided by the present invention;

[0041] Figure 2 A schematic diagram of a matrix block structure provided by the present invention;

[0042] Figure 3 A schematic diagram of the architecture of a block pipeline matrix multiplication provided by the present invention;

[0043] Figure 4 A schematic diagram of the execution of each computing module on an FPGA of a principal component analysis model provided by the present invention;

[0044] Figure 5 A schematic diagram of a framework of hardware-aware automatic quantization in an automatic determination quantization strategy provided by the present invention;

[0045] Figure 6 A schematic diagram of a deployment device for a multimodal remote sensing image interpretation model provided by the present invention;

[0046] Figure 7 A schematic diagram of a computer device for implementing a deployment method of a multimodal remote sensing image interpretation model provided by the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the specific embodiments of the present invention and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0048] Deploying FGPA-GPU heterogeneous computing hardware on UAVs can be applicable to the module tasks of the real-time processing model of multi-modal remote sensing data, and select the most appropriate computing device for calculation, thereby improving the efficiency and performance of the entire system.

[0049] Therefore, based on the multimodal remote sensing data interpretation model, on the FGPA-GPU heterogeneous computing hardware, how to design the optimal acceleration unit for different module tasks and improve the efficiency of synchronous processing and interpretation of different modal data is a key issue in the deployment of multimodal remote sensing fusion lightweight models on heterogeneous computing hardware of unmanned platforms.

[0050] The technical solutions provided by various embodiments of the present invention are described in detail below in conjunction with the accompanying drawings.

[0051] Based on this, the present invention provides a deployment method of a multimodal remote sensing image interpretation model. The technical solutions provided by various embodiments of the present invention are described in detail below in conjunction with the accompanying drawings.

[0052] The execution subject of the deployment method of the multimodal remote sensing image interpretation model in the present invention can be a server, and the server can be a server set up on a business platform, or a device such as a desktop computer, a laptop computer, etc. that can execute the solution of the present invention.

[0053] Figure 1 The following is a flow chart of a deployment method of a multimodal remote sensing image interpretation model in the present invention, which specifically includes the following steps:

[0054] S101, obtaining a benchmark multimodal remote sensing image interpretation model; the benchmark multimodal remote sensing image interpretation model includes a feature extraction model and a feature interpretation model.

[0055] Among them, principal component analysis is a linear transformation, and the transformed data can be divided into multiple principal component components that are linearly independent of each other, and the amount of information contained therein decreases successively. Therefore, the feature extraction model can be a principal component analysis model. The feature extraction model includes an average value calculation module, a covariance matrix calculation module, an eigenvalue decomposition module, a sorting module, and a projection module; therefore, the specific implementation method of the feature extraction model includes the following steps:

[0056] S201, for any multimodal remote sensing image, the multimodal remote sensing image is vectorized by an average value calculation module to obtain an image matrix Multimodal remote sensing images have p bands and a spatial dimension of m×n. is an N-dimensional column vector; and calculate the average value of each row of data in the image matrix X to obtain the mean matrix x ij is the data of the i-th row and j-th column in the matrix, is the average value of all data in the i-th row.

[0057] Where N is the number of pixels in the multimodal remote sensing image.

[0058] S202, through the covariance matrix calculation module, subtract the average value of the same row of data from each data in the image matrix X to obtain the center translation matrix Y: And calculate the covariance matrix of the center translation matrix Y:

[0059] S203, obtaining the eigenvalues ​​of the covariance matrix and the corresponding mutually orthogonal unit eigenvectors through the eigenvalue decomposition module, and forming an eigenvector matrix using the M eigenvectors corresponding to the largest M eigenvalues ​​through the sorting module

[0060] S204, determining the projection matrix according to the eigenvector matrix V and the center translation matrix Y through the projection module

[0061] in, Among them, Z M are the Mth principal components of the multimodal remote sensing images; Z is the new image data matrix that retains the first M principal components after principal component analysis.

[0062] The feature interpretation model is a deep learning model that can analyze the features output by the feature extraction model to obtain the interpretation results of multimodal remote sensing images.

[0063] S102, replacing the matrix multiplication in the feature extraction model with block pipeline matrix multiplication to obtain an optimized feature extraction model; the block pipeline matrix multiplication includes dividing the two matrices to be multiplied into blocks respectively, and processing different block multiplications in sequence according to the block pipeline method.

[0064] The computational complexity and resource consumption of the feature extraction model mainly depend on the size of the input matrix. The consumption of on-chip memory in FPGA is very large and may even exceed the total capacity of the on-chip memory. However, block pipeline matrix multiplication can be used to reduce memory consumption.

[0065] Therefore, the matrix multiplication in the feature extraction model can be replaced by a block pipeline matrix multiplication, including: dividing the center translation matrix Y of size p×N into m blocks in sequence, each block has b rows, m and b satisfy p=m×b, and the transposed matrix of the center translation matrix Y is divided into m blocks, each block has b columns; and processing different block multiplications in sequence in a block pipeline manner to obtain a covariance matrix; finally, updating the feature extraction model according to the calculation method of the covariance matrix.

[0066] Specifically, the main idea of ​​the block pipeline matrix multiplication is to divide the center translation matrix Y of size p×N into m blocks in sequence, with b rows in each block, m and b satisfying p=m×b, and the transposed matrix of Y is also divided into m blocks, with b columns in each block, such as Figure 2 As shown, Figure 2 The figure is a schematic diagram of the structure of matrix block division; two random access memories (RAM) are set up, and the size of each RAM is just enough to store b rows of data. The two RAMs are called RAM1 and RAM2 respectively; data is written into the RAM in blocks. When the data in the RAM is processed, the next block of data is written into the RAM for the next step of processing.

[0067] Here, the m block is divided into three blocks: M1, M2, and M3. Figure 3 As shown, Figure 3 The schematic diagram of the block pipeline matrix multiplication architecture is shown below. Different block multiplications are processed in sequence according to the block pipeline method to obtain the covariance matrix, including:

[0068] S301, the data in M1, M2, and M3 flow into the random access memory RAM1 in blocks in sequence. After the data in the blocks are stored in RAM1, M is calculated and obtained respectively. 11 、M 22 、M 33 .

[0069] S302, control the data in RAM1 to remain unchanged, store the data of M1 and M2 into the random access memory RAM2 in sequence, and calculate M 13 、M 31 、M 23 and M 32 .

[0070] S303, store the data of M2 into RAM1, store the data of M1 into RAM2, and calculate M 12 and M 21 .

[0071] S103, deploying the optimized feature extraction model on the field programmable gate array FPGA in the heterogeneous computing platform of the UAV.

[0072] FPGA has efficient multi-task parallel processing capabilities and is suitable for the synchronous processing of data of different modalities. Therefore, deploying the optimized feature extraction model on FPGA can further improve data processing efficiency.

[0073] The optimized feature extraction model is an optimized principal component analysis model, and the optimized principal component analysis model is deployed on the FPGA of the drone; Figure 4 As shown, Figure 4 This is a schematic diagram of the execution of each computing module of the principal component analysis model on the FPGA; it includes two modules: data scheduling and data processing. First, the data obtained from the memory is allocated to the data processing module through a first-in-first-out queue (FirstInputFirstOutput, FIFO). The data processing module contains multiple submodules. The first is the average value calculation module. The average value vector obtained is stored in the on-chip memory for the subsequent calculation of the covariance matrix and projection. Then there is the covariance matrix calculation module, which obtains the center translation matrix from the average value vector, and then obtains the covariance matrix of the center translation matrix through matrix multiplication. Next is the eigenvalue decomposition module, which uses singular value decomposition instead of eigenvalue decomposition to obtain the eigenvalues ​​and eigenvectors of the covariance matrix. The obtained eigenvalues ​​and eigenvectors are sorted from large to small according to the eigenvalues ​​in the sorting module, and the first M eigenvectors are selected. Finally, in the projection module, the first M eigenvectors selected are multiplied by the center translation matrix respectively to obtain the first M principal component output data.

[0074] S104, quantizing the feature interpretation model, and deploying the quantized feature interpretation model on a graphics processor (GPU) in a heterogeneous computing platform of the UAV.

[0075] Preferably, the feature interpretation model is subjected to model quantization processing, including: determining the quantization bit width of the weights and activation values ​​of each layer in the feature interpretation model based on reinforcement learning and hardware accelerator; updating the feature interpretation model by the quantization bit width of the weights and activation values ​​of each layer to obtain the quantized feature interpretation model.

[0076] Specifically, due to the huge number of parameters in deep learning models, model file storage and operation occupy a large amount of memory overhead. This embodiment designs a hardware-aware autonomous quantization strategy to compress the model parameter volume, reduce the model's computational consumption and improve the model's operating efficiency, so that it can run efficiently under resource-constrained conditions.

[0077] Traditional model quantization methods quantize model weights and activation values ​​to fixed bits. However, due to the characteristics of high-bit quantization with high precision and high consumption and low-bit quantization with low precision and low consumption, it is difficult to achieve the optimal solution between precision and speed under fixed bit quantization. Therefore, this embodiment uses hybrid bit width quantization to further efficiently compress the model, automatically evaluates network complexity and hardware indicators through reinforcement learning, and reasonably allocates the quantization bit width of the weights and activation values ​​of each layer of the model to achieve the best balance between precision and hardware indicators.

[0078] This embodiment designs an automatic determination quantization strategy based on reinforcement learning, obtains feedback from the hardware accelerator in the loop iteration, so that it no longer relies on proxy signals such as floating point operations (FLOP) and model size, but uses a hardware simulator to generate direct feedback signals to the reinforcement learning agent. This quantization strategy is suitable for neural network architectures of different tasks, and can achieve autonomous quantization of multimodal heterogeneous networks, reducing the energy consumption of inference operation with low precision loss.

[0079] The quantization task is modeled as a reinforcement learning problem, such as Figure 5 As shown, Figure 5 Schematic diagram of the framework for hardware-aware automatic quantization in automatic quantization strategy. First, an actor-critic model and a Deep Deterministic Policy Gradient (DDPG) agent are used to give the bits of each layer. Second, hardware counters are collected as constraints and accuracy is used as a reward to search for the optimal quantization strategy. Finally, a hardware environment covering edge, cloud, time and space architectures is built for mixed precision accelerators.

[0080] Specifically, in the state space, a ten-dimensional feature vector is used as the observation value. If the kth layer is a convolutional layer, the state O k for:

[0081] O k =(k,c in ,c out ,s kernel ,s stride ,s feat ,n params ,i dw ,i w / a ,a k-1 ) (1)

[0082] Where k is the layer index, c in is the input channel, c out is the output channel, s kernel is the kernel size, s stride is the step length, s feat is the input feature map size, n params is the parameter, i dw is the binary index of the depthwise convolution, i w / a is a binary indicator of weight / activation, a k-1 is the action at the previous time step.

[0083] If the kth layer is a fully connected layer, state O k for:

[0084] O k =(k,h in ,h out ,1,0,s feat ,n params ,0,i w / a ,a k-1 ) (2)

[0085] Where k is the layer index, h in is the input hidden unit, h out is the output hidden unit, s feat is the input feature map size, n params is the parameter, i w / a is a binary indicator of weight / activation, a k-1 is the action of the previous step. For the observation vector O k Each dimension in is normalized to [0,1] so that they are on the same scale.

[0086] Action Space: Use a continuous action space to determine the bit width. The reason for not using a discrete action space is that it loses the relative order: for example, 2-bit quantization is better than 4-bit or even 8-bit. At the kth time, the continuous action ak (in the range [0, 1]), rounded to a discrete bit width value using the following formula:

[0087] b k =round(b min -0.5+a k ×(b max -b min +1)) (3)

[0088] Resource constraints utilize hardware accelerators to obtain latency and energy as feedback signals to guide the agent to meet resource constraints.

[0089] Quantification method:

[0090] quantize(w,a k ,c)=round(clamp(w,c) / s)×s (4)

[0091] Where s is the scaling factor and clamp is the truncation function. The choice of c is to calculate the KL divergence between the original distribution and the quantized distribution to minimize it. The definition of c is as follows:

[0092]

[0093] Reward function:

[0094]

[0095] Among them, λ can be 0.1, acc quant To evaluate the accuracy of the quantized model, acc origin is the accuracy of the model when not quantized.

[0096] After all layers are quantized, fine-tuning is performed for another epoch, and the validation accuracy after retraining is used as the reward signal.

[0097] The embodiment of the present application also uses RLAgent, which is DDPG, a policy learning method that integrates deep learning neural networks into deep policy gradients. Its Q function setting and loss function setting are:

[0098]

[0099] The loss function is:

[0100]

[0101] Among them, N s Indicates the number of steps, is defined as the exponential moving average of all previous rewards in order to reduce the variance of the gradient estimate. γ is set to 1.

[0102] The quantization bit width of the weights and activation values ​​of each layer in the feature interpretation model is determined by the above-mentioned automatic quantization strategy based on reinforcement learning, and the obtained quantization bit width is used as the quantization bit width of the weights and activation values ​​of each layer in the feature interpretation model to obtain the feature interpretation model after quantization processing.

[0103] In an exemplary embodiment, the method also includes: before model deployment, executing the feature extraction model and the feature interpretation model in the benchmark multimodal remote sensing image interpretation model on different computing units in the heterogeneous computing hardware of the unmanned aerial vehicle, respectively, to obtain the running time of the feature extraction model and the feature interpretation model on the different computing units; the computing units include FPGA and GPU; according to the actual communication rate between the computing units and the running time of each module on the different computing units, determining the optimal computing units for executing the feature extraction model and the feature interpretation model respectively.

[0104] Before the model is deployed on the UAV, the optimal computing unit is designed for measurement. Specifically, a heterogeneous computing platform is deployed on the UAV. The heterogeneous computing platform includes two computing units, FPGA and GPU. In the multi-core system of the heterogeneous platform, the benchmark multimodal remote sensing image interpretation model includes two modules: feature extraction model and feature interpretation model. Due to the different characteristics of the modules and the different structures of the computing units, the time required for various types of computing units to perform the same task is different. In order to measure the consumption of different modules on different computing units, it is planned to run different modules on the target computing unit, record the start time and end time, obtain the module-computing unit time consumption table, and use this as the main basis for allocating model modules.

[0105] At the same time, due to the differences in the computing unit structure of heterogeneous multi-core systems, the connection method needs to consider the speed differences and interface differences of each computing unit, making the connection more complicated. Therefore, it is necessary to calculate the communication overhead between heterogeneous devices, which is determined by the communication rate between computing units and the size of the data dependent between tasks. In order to optimally schedule different computing unit resources, based on the module-computing unit time consumption table, the actual communication rate between computing units is considered, and the task scheduling lookup table is constructed to complete the reasonable mapping between the modules and computing units of the model according to the order of the priority list.

[0106] Through the above-mentioned optimal computing unit calculation, it can be determined that the feature extraction model can be executed on the FPGA, and the feature interpretation model can be executed on the GPU. In this way, the computing flexibility of the GPU and the high throughput performance of the FPGA can be fully utilized, and computing resources can be reasonably allocated according to the different task requirements of multimodal data. The real-time processing capability of remote sensing tasks on the embedded side is improved, thereby improving the interpretation efficiency of the multimodal remote sensing data interpretation model.

[0107] Optionally, before executing the feature extraction model on the FPGA, a noise filtering model may be executed first to filter the multimodal remote sensing image, and features are extracted from the filtered multimodal remote sensing image through the feature extraction model.

[0108] Multimodal data is generally a multidimensional array (i.e., tensor). In this application, in order to improve the operation rate of FPGA on multidimensional data, a corresponding fast tensor operation architecture is designed in the FPGA hardware to parallelize the processing of multidimensional image data. For common basic operations of tensor data, including addition, multiplication, convolution, inner product, outer product, filtering, etc., in order to achieve the efficiency of accelerated parallelization algorithms, it is planned to design and implement the corresponding fast tensor operation architecture according to the operation of basic tensor operations, and encapsulate it in the form of IP core, so as to carry out more complex algorithm design and basic module construction in the future. The design should comply with the large-scale parallel pipeline architecture. The processing unit B passes the computing task to the next processing unit C, and executes the computing task passed by the processing unit A at the same time, forming a pipeline processing architecture. At the same time, there are multiple pipelines for parallel processing, so as to further calculate the tensor fast operation architecture.

[0109] In order to improve the data processing capabilities of edge computing devices, this application adopts a heterogeneous computing mode deployment solution, which can fully utilize the computing flexibility of GPU and the efficiency of FPGA high throughput, reasonably allocate computing resources according to the different task requirements of multimodal data, and improve the real-time processing capabilities of embedded remote sensing tasks. Specifically, the FPGA-GPU heterogeneous computing architecture is integrated to 1) analyze and confirm the optimal acceleration computing unit of different modules; 2) use FPGA to improve the quality and extract features of multimodal data, remove redundant information while retaining key features, and greatly reduce computing costs; 3) design a GPU autonomous perception quantization strategy to quantify the feature interpretation model, and send the extracted features to the GPU for parallel interpretation to achieve efficient execution of data processing and interpretation.

[0110] In an exemplary embodiment, after the multimodal remote sensing image interpretation model is deployed on the UAV, the multimodal remote sensing image to be identified is input into the multimodal remote sensing image interpretation model to obtain the recognition result of the multimodal remote sensing image to be identified.

[0111] When applying the deployment method of the multimodal remote sensing image interpretation model provided by the present invention, it is not necessary to Figure 1 The steps are executed in the order shown. The specific execution order of the steps can be determined according to needs, and the present invention does not limit this.

[0112] The above is a deployment method of a multimodal remote sensing image interpretation model provided by one or more embodiments of the present invention. Based on the same idea, the present invention also provides a corresponding deployment device of a multimodal remote sensing image interpretation model, such as Figure 6 shown.

[0113] Figure 6 A schematic diagram of a deployment device for a multimodal remote sensing image interpretation model provided by the present invention, the device 600 includes:

[0114] An acquisition module 601 is used to acquire a reference multi-modal remote sensing image interpretation model; the reference multi-modal remote sensing image interpretation model includes a feature extraction model and a feature interpretation model;

[0115] An optimization module 602 is used to replace the matrix multiplication in the feature extraction model with a block pipeline matrix multiplication to obtain an optimized feature extraction model; the block pipeline matrix multiplication includes dividing the two matrices to be multiplied into blocks respectively, and processing different block multiplications in sequence according to the block pipeline method;

[0116] The deployment module 603 is used to deploy the optimized feature extraction model on the field programmable gate array FPGA in the heterogeneous computing platform of the drone, and to quantize the feature interpretation model, and deploy the quantized feature interpretation model on the graphics processing unit GPU in the heterogeneous computing platform of the drone.

[0117] For the specific definition of the deployment device of the multimodal remote sensing image interpretation model, please refer to the definition of the deployment method of the multimodal remote sensing image interpretation model in the above text, which will not be repeated here. Each module in the above-mentioned deployment device of the multimodal remote sensing image interpretation model can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0118] The present invention also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 The deployment method of the multimodal remote sensing image interpretation model provided.

[0119] The present invention also provides Figure 7 The structural diagram of the computer device shown in FIG. Figure 7 As shown in the figure, at the hardware level, the computer device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1The deployment method of the multimodal remote sensing image interpretation model provided.

[0120] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0121] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of the present invention.

Claims

1. A method for deploying a multimodal remote sensing image interpretation model, characterized in that: include: Acquire a benchmark multimodal remote sensing image interpretation model; the benchmark multimodal remote sensing image interpretation model includes a feature extraction model and a feature interpretation model; The matrix multiplication in the feature extraction model is replaced by block pipeline matrix multiplication to obtain an optimized feature extraction model; The block pipeline matrix multiplication includes dividing the two matrices to be multiplied into blocks respectively, and processing different block multiplications in sequence in a block pipeline manner; The optimized feature extraction model is deployed on the field programmable gate array FPGA in the heterogeneous computing platform of the drone, and the feature interpretation model is quantized and deployed on the graphics processing unit GPU in the heterogeneous computing platform of the drone.

2. The method according to claim 1, characterized in that The feature extraction model includes an average value calculation module, a covariance matrix calculation module, an eigenvalue decomposition module, a sorting module and a projection module; the specific implementation of the feature extraction model includes: For any multimodal remote sensing image, the multimodal remote sensing image is vectorized through the average value calculation module to obtain the image matrix Multimodal remote sensing images have p bands and a spatial dimension of m×n. is an N-dimensional column vector; calculate the average value of each row of data in the image matrix X to obtain the mean matrix x ij is the data of the i-th row and j-th column in the matrix, is the average value of all data in the i-th row; Through the covariance matrix calculation module, each data in the image matrix X is subtracted from the average value of the same row of data to obtain the center translation matrix Y: And calculate the covariance matrix of the center translation matrix Y: The eigenvalue decomposition module is used to obtain the eigenvalues ​​of the covariance matrix and the corresponding mutually orthogonal unit eigenvectors, and the sorting module is used to form an eigenvector matrix using the M eigenvectors corresponding to the largest M eigenvalues. The projection matrix is ​​determined by the projection module based on the eigenvector matrix V and the center translation matrix Y.

3. The method according to claim 2, characterized in that The step of replacing the matrix multiplication in the feature extraction model with the block pipeline matrix multiplication includes: The center translation matrix Y of size p×N is divided into m blocks in sequence, each block has b rows, m and b satisfy p=m×b, and the transposed matrix of the center translation matrix Y is divided into m blocks, each block has b columns; Process different block multiplications in sequence according to the block pipeline method to obtain the covariance matrix; Update the feature extraction model based on how the covariance matrix is ​​calculated.

4. The method according to claim 3, characterized in that: The m blocks include M1, M2, and M3; the different block multiplications are processed in sequence in a block pipeline manner to obtain a covariance matrix, including: The data in M1, M2, and M3 flow into the random access memory RAM1 in blocks. After the data in the blocks are stored in RAM1, M is calculated and obtained respectively. 11 、M 22 、M 33 ; Control the data in RAM1 to remain unchanged, store the data of M1 and M2 into the random access memory RAM2 in turn, and calculate M 13 、M 31 、M 23 and M 32 ; Store the data of M2 in RAM1 and the data of M1 in RAM2, and calculate M 12 and M 21 .

5. The method according to any one of claims 1 to 4, characterized in that: The step of performing model quantization processing on the feature interpretation model includes: Determine the quantization bit width of weights and activation values ​​of each layer in the feature interpretation model based on reinforcement learning and hardware accelerators; The feature interpretation model is updated by the quantization bit width of the weights and activation values ​​of each layer to obtain a quantized feature interpretation model.

6. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Before model deployment, the feature extraction model and feature interpretation model in the benchmark multimodal remote sensing image interpretation model are respectively executed on different computing units in the heterogeneous computing hardware of the UAV to obtain the running time of the feature extraction model and the feature interpretation model on different computing units; the computing units include FPGA and GPU; According to the actual communication rate between the computing units and the running time of each module executed on different computing units, the optimal computing units for executing the feature extraction model and the feature interpretation model are determined respectively.

7. A deployment device for a multimodal remote sensing image interpretation model, characterized in that: include: An acquisition module is used to acquire a benchmark multimodal remote sensing image interpretation model; the benchmark multimodal remote sensing image interpretation model includes a feature extraction model and a feature interpretation model; An optimization module, used to replace the matrix multiplication in the feature extraction model with a block pipeline matrix multiplication to obtain an optimized feature extraction model; The block pipeline matrix multiplication includes dividing the two matrices to be multiplied into blocks respectively, and processing different block multiplications in sequence in a block pipeline manner; The deployment module is used to deploy the optimized feature extraction model on the field programmable gate array FPGA in the heterogeneous computing platform of the drone, and to quantize the feature interpretation model, and then deploy the quantized feature interpretation model on the graphics processing unit GPU in the heterogeneous computing platform of the drone.

8. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

9. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method according to any one of claims 1 to 6 is implemented.

Images