Multi-view target image fusion method, device and equipment and readable storage medium

Abstract

The invention discloses a multi-view target image fusion method, device and equipment and a readable storage medium, and the method comprises the steps: carrying out the spatial position coding of the azimuth angle, pitch angle and distance of each image relative to a shooting position, and splicing the obtained spatial position code of each image with a corresponding image block embedding vector; forming a spatial image block embedding sequence by using all the spatial image block embedding vectors, and adding a learnable classification mark at the foremost end of the spatial image block embedding sequence to obtain a new spatial image block embedding sequence; inputting the new space image block embedding sequence into a Transform model, extracting global information in all space image block embedding vectors based on a proxy attention mechanism to update learnable classification marks, and obtaining a fused classification mark; and performing category prediction on the target through a multi-layer perceptron by using the fused classification mark. According to the invention, the target identification accuracy of the model can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of image target recognition, and in particular to a multi-view target image fusion method, apparatus, device and readable storage medium. Background Art

[0002] Existing multi-view object recognition methods use multiple cameras to capture the target from different angles, improving detection accuracy and reducing occlusion issues associated with a single viewpoint. Deep learning object detection algorithms such as YOLO and DETR are typically used to achieve single-machine object recognition. Feature-level fusion or decision-level fusion is then performed based on the single-machine recognition results to achieve joint recognition. Feature-level fusion extracts features from the images detected by a single machine and then fuses the features. Decision-level fusion involves each device independently detecting the target and ultimately merging the decisions through methods such as voting and Bayesian reasoning.

[0003] ViT (Vision Transformer) is a computer vision model based on the Transformer architecture. It breaks away from the traditional reliance on convolutional operations for feature extraction. Instead, it employs a self-attention mechanism to fuse features from multiple image patches in a single machine image for global feature modeling, resulting in outstanding performance on large-scale image recognition datasets. Transformer is a deep learning architecture that leverages its self-attention mechanism and parallel computing to better handle long-range dependencies, improving model training and inference efficiency.

[0004] However, the current ViT model still performs poorly for multi-view object recognition. Summary of the Invention

[0005] The present application provides a multi-view target image fusion method, apparatus, device and readable storage medium, aiming to solve the technical problem that the current ViT model is still not effective for multi-view target recognition.

[0006] In a first aspect, an embodiment of the present application provides a multi-view target image fusion method, the multi-view target image fusion method comprising: Acquire multiple images of the target from multiple perspectives, and obtain the azimuth, pitch angle, and distance of each image relative to the shooting position; Use the target detection algorithm to extract the image blocks of each image, transform the image blocks of each image through the linear mapping layer, and obtain the image block embedding vector of each image; The azimuth, pitch angle, and distance of each image relative to the shooting position are spatially encoded, and the obtained spatial position encoding of each image is spliced ​​with the corresponding image block embedding vector to obtain the spatial image block embedding vector of each image; The spatial image block embedding sequence is composed of all spatial image block embedding vectors, and a learnable classification tag is added to the front end of the spatial image block embedding sequence to obtain a new spatial image block embedding sequence; The new spatial image patch embedding sequence is input into the Transformer model. Based on the proxy attention mechanism, the global information in the embedding vectors of all spatial image patches is extracted to update the learnable classification labels to obtain the fused classification labels. The fused classification labels are used to predict the target category through a multi-layer perceptron.

[0007] Optionally, extracting an image block from each image using an object detection algorithm includes: For each image, use the object detection algorithm to identify the object in the image; If the target in the image is recognized, the target area in the image is extracted to obtain an image block; If the target in the image is not recognized, a mask image block is generated as the extracted image block.

[0008] Optionally, the spatial position encoding of the azimuth, pitch angle, and distance of each image relative to the shooting position includes: Normalize the azimuth, elevation, and distance of each image relative to the shooting position to obtain the normalized spatial position vector of each image; The normalized spatial position vector of each image is transformed through a linear mapping layer to obtain the spatial position encoding of each image.

[0009] Optionally, the new spatial image patch embedding sequence is input into the Transformer model, and the global information in all spatial image patch embedding vectors is extracted based on the proxy attention mechanism to update the learnable classification labels. The obtained fused classification labels include: The new spatial image patch embedding sequence is input into the Transformer model, and the learnable classification label and each spatial image patch embedding vector are mapped into the corresponding query vector, key vector and value vector respectively through the learnable linear transformation of the Transformer model; Calculate the proxy score for each query vector through the perceptron; Select a preset number of query vectors with the highest proxy scores from all query vectors as proxy vectors; Taking each proxy vector as a query, perform attention calculation with each key vector and value vector respectively to obtain the converged features of each proxy vector; Taking the query vector corresponding to the learnable classification label as the query, perform attention calculation with the aggregated features of each proxy vector, and update the value vector corresponding to the learnable classification label; The value vector corresponding to the updated learnable classification label is used as the fused classification label.

[0010] Optionally, the output result of the target category prediction performed by the multilayer perceptron includes the target category, the target position, and the target confidence. After the fused classification label is used to predict the target category by the multilayer perceptron, the following steps are included: Visualize the output of target category, target location and target confidence.

[0011] In a second aspect, an embodiment of the present application provides a multi-view target image fusion device, the multi-view target image fusion device comprising: An acquisition module is used to acquire multiple images of a target from multiple perspectives, and obtain the azimuth, pitch angle and distance of each image relative to the shooting position; A conversion module is used to extract image blocks of each image using an object detection algorithm, and transform the image blocks of each image through a linear mapping layer to obtain an image block embedding vector for each image; A splicing module is used to perform spatial position encoding on the azimuth, pitch angle, and distance of each image relative to the shooting position, and to splice the obtained spatial position encoding of each image with the corresponding image block embedding vector to obtain the spatial image block embedding vector of each image; Add a module for forming a spatial image block embedding sequence with all spatial image block embedding vectors, and adding a learnable classification tag at the front end of the spatial image block embedding sequence to obtain a new spatial image block embedding sequence; The fusion module is used to input the new spatial image patch embedding sequence into the Transformer model, extract the global information in all spatial image patch embedding vectors based on the proxy attention mechanism, update the learnable classification label, and obtain the fused classification label; The prediction module is used to use the fused classification labels to predict the target category through a multi-layer perceptron.

[0012] Optionally, the spatial position encoding of the azimuth, elevation, and distance of each image relative to the shooting position is used for: Normalize the azimuth, elevation, and distance of each image relative to the shooting position to obtain the normalized spatial position vector of each image; The normalized spatial position vector of each image is transformed through a linear mapping layer to obtain the spatial position encoding of each image.

[0013] Optionally, the fusion module is used to: The new spatial image patch embedding sequence is input into the Transformer model, and the learnable classification label and each spatial image patch embedding vector are mapped into the corresponding query vector, key vector and value vector respectively through the learnable linear transformation of the Transformer model; Calculate the proxy score for each query vector through the perceptron; Select a preset number of query vectors with the highest proxy scores from all query vectors as proxy vectors; Taking each proxy vector as a query, perform attention calculation with each key vector and value vector respectively to obtain the converged features of each proxy vector; Taking the query vector corresponding to the learnable classification label as the query, perform attention calculation with the aggregated features of each proxy vector, and update the value vector corresponding to the learnable classification label; The value vector corresponding to the updated learnable classification label is used as the fused classification label.

[0014] In a third aspect, an embodiment of the present application provides a multi-perspective target image fusion device, which includes a processor, a memory, and a multi-perspective target image fusion program stored on the memory and executable by the processor, wherein when the multi-perspective target image fusion program is executed by the processor, the steps of the multi-perspective target image fusion method described above are implemented.

[0015] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a multi-perspective target image fusion program is stored, wherein when the multi-perspective target image fusion program is executed by a processor, the steps of the multi-perspective target image fusion method described above are implemented.

[0016] The beneficial effects of the technical solutions provided in the embodiments of the present application include: In an embodiment of the present application, multiple images of a target are obtained by photographing the target from multiple perspectives, and the azimuth, pitch angle and distance of each image relative to the shooting position are obtained; an image block of each image is extracted using a target detection algorithm, and the image block of each image is converted through a linear mapping layer to obtain an image block embedding vector for each image; the azimuth, pitch angle and distance of each image relative to the shooting position are spatially encoded, and the obtained spatial position encoding of each image is spliced ​​with the corresponding image block embedding vector to obtain a spatial image block embedding vector for each image; a spatial image block embedding sequence is formed by all spatial image block embedding vectors, and a learnable classification tag is added to the front end of the spatial image block embedding sequence to obtain a new spatial image block embedding sequence; the new spatial image block embedding sequence is input into a Transformer model, and the global information in all spatial image block embedding vectors is extracted based on the proxy attention mechanism to update the learnable classification tag to obtain a fused classification tag; the fused classification tag is used to predict the category of the target through a multi-layer perceptron. Through the embodiments of the present application, by adding spatial position encoding to the image block embedding vectors of each image of multiple perspectives, the Transformer model can better fuse the image visual information of the target and the spatial position information of the target, thereby improving the accuracy of the model in identifying the target. The proxy attention mechanism of the Transformer model can reduce the number of queries involved in the calculation, thereby greatly improving the operating efficiency of the model. The learnable classification tag is a vector that can be randomly initialized, learns to change during the fusion process, and is ultimately used for perceptron classification prediction. The learnable classification tag is added at the forefront of the spatial image block embedding sequence. The learnable classification tag can more conveniently interact with all spatial image block embedding vectors through the proxy attention mechanism, learn the global information in all spatial image block embedding vectors, and efficiently fuse the global features of images from multiple perspectives. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a flow chart of an embodiment of the multi-view target image fusion method of the present application; Figure 2 For this application Figure 1 Detailed flow chart of step S50; Figure 3 This is another flowchart of an embodiment of the multi-view target image fusion method of the present application; Figure 4 This is a schematic diagram of the functional modules of an embodiment of the multi-view target image fusion device of the present application; Figure 5 This is a schematic diagram of the hardware structure of the multi-view target image fusion device involved in the embodiment of the present application. DETAILED DESCRIPTION

[0018] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0019] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0020] In a first aspect, an embodiment of the present application provides a multi-view target image fusion method.

[0021] In one embodiment, referring to Figure 1 , Figure 1 This is a flow chart of an embodiment of the multi-view target image fusion method of the present application. Figure 1 As shown, the multi-view target image fusion method includes: Step S10 , obtaining a plurality of images obtained by photographing the target from a plurality of viewing angles, and obtaining the azimuth angle, pitch angle, and distance of each image relative to the photographing position.

[0022] In this embodiment, for example, in anti-UAV application scenarios, in order to achieve all-round monitoring and accurate identification of aerial targets, a multi-camera collaborative acquisition solution can be adopted, and multiple high-definition cameras are installed on multiple carrier aircraft (such as UAVs or fixed-wing aircraft) to ensure that each carrier aircraft is equipped with at least one camera. The carrier aircraft dynamically flies around the target UAV according to a preset flight path to form a multi-perspective observation network. The flight path of the carrier aircraft can be designed as a circular or spiral trajectory around the target UAV to ensure that the camera collects all-round video image data of the target from different perspectives (such as looking down, looking up, and looking to the side, etc.), wherein the perspectives between the carrier aircraft are evenly distributed to avoid perspective overlap or blind spots.

[0023] Specifically, for example, three drones equipped with high-definition cameras capture a target (such as a drone or bird) at different viewing angles (0°, 60°, and 120°). Each drone captures five images, resulting in a total of 15 images of the target from multiple perspectives. The following parameters are collected for each image: Azimuth: the horizontal angle of the target relative to the drone (range 0° to 360°); Elevation: the vertical angle of the target relative to the drone (range -90° to 90°); and Distance: the Euclidean distance between the target and the drone (which can be obtained using lidar or visual depth estimation).

[0024] In step S20 , an image block of each image is extracted using an object detection algorithm, and the image block of each image is transformed through a linear mapping layer to obtain an image block embedding vector of each image.

[0025] In this embodiment, the YOLOV8 target detection algorithm can be used to process each image of multiple perspectives separately, locate the target area, generate a bounding box, and extract the image area including the target part based on the bounding box to obtain the image block of each image, which can be scaled to a fixed size (for example, 16×16 pixels). The image block of each image is converted into a 256-dimensional image block embedding vector through a linear mapping layer. The image block of each image is converted into a high-dimensional feature vector through a linear mapping layer because, in the target detection and recognition process, the original pixel data of the image block (target area) has a low dimension (for example, 16×16 pixels). Directly inputting it into the Transformer model may result in loss of semantic information due to insufficient feature dimension. The role of the linear mapping layer is to convert the low-dimensional image block data into a high-dimensional feature vector to meet the needs of the subsequent feature fusion module.

[0026] Step S30 , spatially encoding the azimuth, pitch angle, and distance of each image relative to the shooting position, and concatenating the obtained spatial position encoding of each image with the corresponding image block embedding vector to obtain a spatial image block embedding vector for each image.

[0027] In this embodiment, by introducing spatial parameters such as the azimuth, pitch angle and distance of the target, the azimuth, pitch angle and distance of each image relative to the shooting position are spatially encoded, and the spatial position encoding of each image is spliced ​​with the corresponding image block embedding vector, so that the spatial image block embedding vector of each image includes both the image data itself and the spatial position information, thereby enabling the model to perceive the geometric spatial relationship between image blocks, enhancing the model's understanding of the target position and motion trajectory, and improving the model's accuracy in target recognition.

[0028] In step S40 , all spatial image block embedding vectors are used to form a spatial image block embedding sequence, and a learnable classification tag is added to the front end of the spatial image block embedding sequence to obtain a new spatial image block embedding sequence.

[0029] In this embodiment, the learnable classification tag is a vector that can be randomly initialized, learns and changes during the fusion process, and is ultimately used for perceptron classification prediction. The learnable classification tag is added at the front end of the spatial image block embedding sequence. The learnable classification tag can more conveniently interact with all spatial image block embedding vectors through the proxy attention mechanism, learn the global information in all spatial image block embedding vectors, and efficiently fuse the global features of multiple-view images. Continuing with the example of 15 images from multiple perspectives, the new spatial image block embedding sequence can be expressed as NewSequence=[CLS]||X1||X2||...||X 15 , where [CLS] represents the learnable classification label, X i Represents the embedding vector of each spatial image block, i represents the number of all spatial image block embedding vectors, i=15, where each spatial image block embedding vector includes a spatial position code and an image block embedding vector.

[0030] In step S50, the new spatial image block embedding sequence is input into the Transformer model, and the global information in the embedding vectors of all spatial image blocks is extracted based on the proxy attention mechanism to update the learnable classification labels to obtain the fused classification labels.

[0031] In this embodiment, the proxy attention mechanism is an optimization or approximation of the standard self-attention mechanism, designed to significantly reduce computational complexity. Its core is the introduction of a set of learnable proxy vectors whose number is much smaller than the length of the input sequence. The number of proxy vectors is much smaller than the number of queries, significantly reducing computational complexity. Through indirect interaction with the proxy vectors, the receptive field limitations of local window attention can be avoided, supporting global context modeling. Continuing with the example of 15 images from multiple perspectives, the classifiable learning tag at the front end of the new spatial image block embedding sequence interacts with each spatial image block embedding vector through the learnable classification tag [CLS] of the proxy attention mechanism, extracting global information from each spatial image block embedding vector and updating the learnable classification tag. After fusion through a multi-layer network, the fused classification tag is obtained.

[0032] Step S60: Use the fused classification labels to predict the category of the target through a multi-layer perceptron.

[0033] In this embodiment, a multilayer perceptron (MLP) is a classic feedforward neural network, typically composed of multiple fully connected layers and nonlinear activation functions (such as ReLU). At the output layer of the MLP, a Softmax function is used to calculate the probability of each category, and the category with the highest probability is selected as the final prediction result. Specifically, the fused classification labels serve as input to the MLP, and the MLP output includes, for example, the target category (e.g., the predicted target category is a drone), the target location (the target's spatial location), and the target confidence score (e.g., 0.95).

[0034] In this embodiment, a carrier aircraft equipped with a camera can be used to collect multiple images of the target from multiple perspectives, as well as the azimuth, pitch angle, and distance of each image relative to the shooting position. The YOLOV8 target detection algorithm is used to process each image from multiple perspectives separately, and the image block of each image is converted into a high-dimensional feature vector through a linear mapping layer, and the low-dimensional image block data is converted into a high-dimensional feature vector, which can better meet the needs of the subsequent feature fusion module. By introducing spatial parameters such as the azimuth, pitch angle, and distance of the target, the azimuth, pitch angle, and distance of each image relative to the shooting position are spatially encoded, and the spatial position encoding of each image is spliced ​​with the corresponding image block embedding vector, so that the spatial image block embedding vector of each image includes both the image data itself and the spatial position information, thereby enabling the model to perceive the geometric spatial relationship between image blocks, enhancing the model's understanding of the target position and motion trajectory, and improving the model's accuracy in target recognition. The learnable classification tag is a vector that can be randomly initialized, learns to change during the fusion process, and is ultimately used for perceptron classification prediction. Adding the learnable classification tag to the front of the spatial image patch embedding sequence allows it to more conveniently interact with all spatial image patch embeddings through a proxy attention mechanism, learning global information from all spatial image patch embeddings and efficiently fusing global features from multiple viewpoints. Through the proxy attention mechanism, the learnable classification tag interacts with each spatial image patch embedding, extracting global information from each embedding and updating the learnable classification tag. After fusion through a multi-layer network, the fused classification tag is obtained. The proxy attention mechanism significantly reduces the model's computational complexity and improves its operational efficiency. By considering not only the similarity of visual features but also the physical spatial position of the target when calculating the correlation between multiple viewpoints, the proposed method enhances the robustness of the fusion process and significantly improves the feature discrimination capability of multi-viewpoint images, especially in complex scenarios such as target occlusion and pose changes. It has broad applications in fields such as drone monitoring, air traffic management, and military reconnaissance.

[0035] Furthermore, in one embodiment, extracting image blocks from each image using an object detection algorithm includes: For each image, use the object detection algorithm to identify the object in the image; If the target in the image is recognized, the target area in the image is extracted to obtain an image block; If the target in the image is not recognized, a mask image block is generated as the extracted image block.

[0036] In this embodiment, for each image, the YOLOV8 target detection algorithm can be used to identify the target in the image, which can be processed in two situations. One is that the target is not obscured, and the YOLOV8 target detection algorithm identifies the target in the image, then locates the target area, generates a bounding box, and extracts the image area including the target part according to the bounding box to obtain the image block of the image (which can be scaled to a fixed size, such as 16×16 pixels). The other is that the target is obscured, and the YOLOV8 target detection algorithm does not identify the target in the image, then generates a mask image block (an all-zero matrix 16×16 can be used, which is consistent with the size of the image block extracted when the target is identified) as the extracted image block to maintain data consistency.

[0037] Furthermore, in one embodiment, the spatial position encoding of the azimuth, pitch angle, and distance of each image relative to the shooting position includes: Normalize the azimuth, elevation, and distance of each image relative to the shooting position to obtain the normalized spatial position vector of each image; The normalized spatial position vector of each image is transformed through a linear mapping layer to obtain the spatial position encoding of each image.

[0038] In this embodiment, the shooting position parameters of each captured image are: Azimuth: the horizontal angle of the target relative to the drone (range 0°~360°), Elevation: the vertical angle of the target relative to the drone (range -90°~90°), Distance: the Euclidean distance between the target and the drone (obtained by lidar or visual depth estimation). The azimuth, elevation, and distance are normalized to the range [0,1] using the formula, for example: , , , where Azimuth, Elevation and Distance represent the collected azimuth, elevation and distance respectively. norm 、Elevation norm and Distance normwhere represents the corresponding normalized value, and Dmax represents the maximum detection distance (e.g., 500 meters). Specifically, the normalized azimuth, elevation, and distance of an image are, for example, [0.25, 0.5, 0.1]. These are transformed through the linear mapping layer to obtain the spatial position encoding.

[0039] Furthermore, in one embodiment, referring to Figure 2 , Figure 2 For this application Figure 1 The detailed flow chart of step S50 is as follows: Figure 2 As shown, step S50 includes: Step S501: Input the new spatial image patch embedding sequence into the Transformer model, and use the learnable linear transformation of the Transformer model to map the learnable classification label and each spatial image patch embedding vector into the corresponding query vector, key vector, and value vector respectively; Step S502, calculating the proxy score of each query vector by using a perceptron; Step S503 , selecting a preset number of query vectors with the highest proxy scores from all query vectors as proxy vectors; Step S504: Taking each proxy vector as a query, perform attention calculation with each key vector and value vector respectively to obtain the converged features of each proxy vector; Step S505 , using the query vector corresponding to the learnable classification label as a query, performing attention calculation on the converged features of each proxy vector, and updating the value vector corresponding to the learnable classification label; Step S506: Use the value vector corresponding to the updated learnable classification label as the fused classification label.

[0040] In this embodiment, taking 15 images from multiple perspectives as an example, the new spatial image block embedding sequence is NewSequence=[CLS]||X1||X2||...||X 15 , the new spatial image block embedding sequence NewSequence is input into the Transformer model, and each spatial image block is embedded into the vector X through the learnable linear transformation matrix of the Transformer model such as W_Q, W_K and W_V iAnd the learnable classification tag [CLS] is mapped to the corresponding query vector Query, key vector Key and value vector Value respectively. The proxy score of each query vector Query is calculated by the perceptron (a lightweight two-layer perceptron can be used). The proxy score can reflect the representativeness of the query vector Query in the global context. The higher the proxy score, the more important the query vector Query is and the more suitable it is to be used as a proxy. All query vectors are sorted according to the proxy score, and a preset number of query vectors with the highest proxy scores are selected from all query vectors as proxy vectors. For example, the five query vectors with the highest proxy scores are selected as proxy vectors, namely Q1, Q3, Q5, Q7 and Q9, to form a proxy vector set A. j .

[0041] Then, the first attention calculation is performed to extract the global information from all key vectors and value vectors, and each proxy vector is used as a query to compare it with all key vectors K. i Sum value vector V i Calculate the similarity, including all image block embedding vectors and key vectors and value vectors corresponding to the learnable classification tags. Each proxy vector A j Generate the corresponding convergence feature VA j , the formula for the first attention calculation can be expressed as , where d k is the dimension of the key vector, and the superscript T represents the key vector K i The transposed matrix of , used to provide the correct dimension matching for matrix multiplication.

[0042] Then, a second attention calculation is performed to update the global information in all key vectors and value vectors learned through the proxy vector to the value vector of the classifiable label, and the query vector Q corresponding to the learnable classification label [CLS] CLS For query, the aggregated feature VA of each proxy vector j Perform attention calculation and update the value vector corresponding to the learnable classification label. The formula for the second attention calculation can be expressed as , among which, VA j T The Key vector calculated for the first attention (the Key of the proxy vector), VA jThe value vector (the value of the proxy vector) calculated for the first attention step allows the learnable classification label [CLS] to interact with all key and value vectors through the proxy vector. The learnable classification label [CLS] thus learns the global information in the embedding vectors of all spatial image patches. The proxy vector significantly reduces the model's computational complexity and improves its operational efficiency. The proxy score effectively utilizes global features from high-scoring perspective images. The fused classification label is obtained through multi-layer network fusion, where the input of each network layer is the output of the previous layer.

[0043] Furthermore, in one embodiment, the output result of the multi-layer perceptron for target category prediction includes target category, target location and target confidence. Figure 3 , Figure 3 This is another flowchart of an embodiment of the multi-view target image fusion method of the present application. Figure 3 As shown, after step S60, the following steps are included: Step S70: Visually output the target category, target location, and target confidence.

[0044] In this embodiment, for example, the target category, target location and target confidence predicted by the model, as well as the real-time video image of the target are displayed in real time on the monitoring terminal, providing a visual interface and structured data interface to support real-time monitoring and decision-making.

[0045] In a second aspect, an embodiment of the present application also provides a multi-view target image fusion device.

[0046] In one embodiment, referring to Figure 4 , Figure 4 This is a functional module diagram of an embodiment of the multi-view target image fusion device of the present application, as shown in FIG. Figure 4 As shown, the multi-view target image fusion device includes: An acquisition module 10 is used to acquire multiple images of a target from multiple perspectives, and obtain the azimuth, pitch angle, and distance of each image relative to the shooting position; A conversion module 20 is configured to extract image blocks from each image using an object detection algorithm, and convert the image blocks from each image through a linear mapping layer to obtain an image block embedding vector for each image; a stitching module 30 for encoding the azimuth, elevation, and distance of each image relative to the shooting position, and stitching the obtained spatial position encoding of each image with the corresponding image block embedding vector to obtain a spatial image block embedding vector for each image; An adding module 40 is configured to form a spatial image block embedding sequence using all spatial image block embedding vectors, and to add a learnable classification tag at the front end of the spatial image block embedding sequence to obtain a new spatial image block embedding sequence; A fusion module 50 is configured to input the new spatial image patch embedding sequence into the Transformer model, extract the global information in all spatial image patch embedding vectors based on the proxy attention mechanism, update the learnable classification labels, and obtain the fused classification labels; The prediction module 60 is used to use the fused classification labels to predict the category of the target through a multi-layer perceptron.

[0047] Furthermore, in one embodiment, extracting image blocks from each image using an object detection algorithm includes: For each image, use the object detection algorithm to identify the object in the image; If the target in the image is recognized, the target area in the image is extracted to obtain an image block; If the target in the image is not recognized, a mask image block is generated as the extracted image block.

[0048] Furthermore, in one embodiment, the spatial position encoding of the azimuth, pitch angle, and distance of each image relative to the shooting position is used to: Normalize the azimuth, elevation, and distance of each image relative to the shooting position to obtain the normalized spatial position vector of each image; The normalized spatial position vector of each image is transformed through a linear mapping layer to obtain the spatial position encoding of each image.

[0049] Furthermore, in one embodiment, the fusion module 50 is configured to: The new spatial image patch embedding sequence is input into the Transformer model, and the learnable classification label and each spatial image patch embedding vector are mapped into the corresponding query vector, key vector and value vector respectively through the learnable linear transformation of the Transformer model; Calculate the proxy score for each query vector through the perceptron; Select a preset number of query vectors with the highest proxy scores from all query vectors as proxy vectors; Taking each proxy vector as a query, perform attention calculation with each key vector and value vector respectively to obtain the converged features of each proxy vector; Taking the query vector corresponding to the learnable classification label as the query, perform attention calculation with the aggregated features of each proxy vector, and update the value vector corresponding to the learnable classification label; The value vector corresponding to the updated learnable classification label is used as the fused classification label.

[0050] Furthermore, in one embodiment, the output result of the target category prediction performed by the multi-layer perceptron includes the target category, target position, and target confidence, and the multi-view target image fusion device further includes an output module for: Visualize the output of target category, target location and target confidence.

[0051] Among them, the functional implementation of each module in the above-mentioned multi-view target image fusion device corresponds to the various steps in the above-mentioned multi-view target image fusion method embodiment, and their functions and implementation processes are no longer repeated here.

[0052] In a third aspect, an embodiment of the present application provides a multi-perspective target image fusion device.

[0053] Reference Figure 5 , Figure 5 FIG2 is a schematic diagram of the hardware structure of a multi-view target image fusion device involved in an embodiment of the present application. In the embodiment of the present application, the multi-view target image fusion device may include a processor, a memory, a communication interface, and a communication bus.

[0054] The communication bus may be of any type and is used to interconnect the processor, memory, and communication interface.

[0055] Communication interfaces include input / output (I / O), physical, and logical interfaces, which interconnect components within the multi-view target image fusion device and connect it to other devices (such as other computing devices or user devices). Physical interfaces can include Ethernet, fiber, and ATM interfaces; user devices can include displays and keyboards.

[0056] The memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.

[0057] The processor may be a general-purpose processor that can invoke a multi-view target image fusion program stored in a memory and execute the multi-view target image fusion method provided in the embodiments of the present application. For example, the general-purpose processor may be a central processing unit (CPU). The method executed when the multi-view target image fusion program is invoked can be referenced from the various embodiments of the multi-view target image fusion method of the present application and will not be further described here.

[0058] Those skilled in the art will understand that Figure 5 The hardware structure shown in the figure does not constitute a limitation to the present application and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0059] In a fourth aspect, an embodiment of the present application also provides a readable storage medium.

[0060] The readable storage medium of the present application stores a multi-view target image fusion program, wherein when the multi-view target image fusion program is executed by a processor, the steps of the multi-view target image fusion method as described above are implemented.

[0061] Among them, the method implemented when the multi-view target image fusion program is executed can refer to the various embodiments of the multi-view target image fusion method of this application, and will not be repeated here.

[0062] It should be noted that the serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0063] The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices. The terms "first", "second" and "third" are used to distinguish different objects, etc., and do not represent a sequence, nor do they limit the "first", "second" and "third" to different types.

[0064] In the description of the embodiments of this application, the words "exemplary," "for example," or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary," "for example," or "for example" in the embodiments of this application should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary," "for example," or "for example" is intended to present the relevant concepts in a concrete manner.

[0065] In the description of the embodiments of the present application, unless otherwise specified, “ / ” means or, for example, A / B can mean A or B; “and / or” in the text is merely a description of the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, “multiple” refers to two or more than two.

[0066] In some processes described in the embodiments of the present application, multiple operations or steps are included that appear in a specific order. However, it should be understood that these operations or steps may not be performed in the order in which they appear in the embodiments of the present application or may be performed in parallel. The sequence numbers of the operations are only used to distinguish between different operations, and the sequence numbers themselves do not represent any order of execution. In addition, these processes may include more or fewer operations, and these operations or steps may be performed in sequence or in parallel, and these operations or steps may be combined.

[0067] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, or the part that contributes to the existing technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above and includes a number of instructions for enabling a terminal device to execute the methods described in each embodiment of this application.

[0068] The above are only preferred embodiments of the present application and do not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A multi-view target image fusion method, characterized in that: The multi-view target image fusion method comprises: Acquire multiple images of the target from multiple perspectives, and obtain the azimuth, pitch angle, and distance of each image relative to the shooting position; Use the target detection algorithm to extract the image blocks of each image, transform the image blocks of each image through the linear mapping layer, and obtain the image block embedding vector of each image; The azimuth, pitch angle, and distance of each image relative to the shooting position are spatially encoded, and the obtained spatial position encoding of each image is spliced ​​with the corresponding image block embedding vector to obtain the spatial image block embedding vector of each image; The spatial image block embedding sequence is composed of all spatial image block embedding vectors, and a learnable classification tag is added to the front end of the spatial image block embedding sequence to obtain a new spatial image block embedding sequence; The new spatial image patch embedding sequence is input into the Transformer model. Based on the proxy attention mechanism, the global information in the embedding vectors of all spatial image patches is extracted to update the learnable classification labels to obtain the fused classification labels. The fused classification labels are used to predict the target category through a multi-layer perceptron.

2. The multi-view target image fusion method according to claim 1, wherein: The extracting of image blocks from each image using a target detection algorithm includes: For each image, use the object detection algorithm to identify the object in the image; If the target in the image is recognized, the target area in the image is extracted to obtain an image block; If the target in the image is not recognized, a mask image block is generated as the extracted image block.

3. The multi-view target image fusion method according to claim 1, wherein: The spatial position encoding of the azimuth, pitch angle and distance of each image relative to the shooting position includes: Normalize the azimuth, elevation, and distance of each image relative to the shooting position to obtain the normalized spatial position vector of each image; The normalized spatial position vector of each image is transformed through a linear mapping layer to obtain the spatial position encoding of each image.

4. The multi-view target image fusion method according to claim 1, wherein: The new spatial image block embedding sequence is input into the Transformer model, and the global information in all spatial image block embedding vectors is extracted based on the proxy attention mechanism to update the learnable classification labels. The fused classification labels include: The new spatial image patch embedding sequence is input into the Transformer model, and the learnable classification label and each spatial image patch embedding vector are mapped into the corresponding query vector, key vector and value vector respectively through the learnable linear transformation of the Transformer model; Calculate the proxy score for each query vector through the perceptron; Select a preset number of query vectors with the highest proxy scores from all query vectors as proxy vectors; Taking each proxy vector as a query, perform attention calculation with each key vector and value vector respectively to obtain the converged features of each proxy vector; Taking the query vector corresponding to the learnable classification label as the query, perform attention calculation with the aggregated features of each proxy vector, and update the value vector corresponding to the learnable classification label; The value vector corresponding to the updated learnable classification label is used as the fused classification label.

5. The multi-view target image fusion method according to claim 1, wherein: The output result of the multilayer perceptron for target category prediction includes target category, target location and target confidence. After the fused classification label is used to predict the target category through the multilayer perceptron, the following is included: Visualize the output of target category, target location and target confidence.

6. A multi-view target image fusion device, characterized in that: The multi-view target image fusion device comprises: An acquisition module is used to acquire multiple images of a target from multiple perspectives, and obtain the azimuth, pitch angle and distance of each image relative to the shooting position; A conversion module is used to extract image blocks of each image using an object detection algorithm, and transform the image blocks of each image through a linear mapping layer to obtain an image block embedding vector for each image; A splicing module is used to perform spatial position encoding on the azimuth, pitch angle, and distance of each image relative to the shooting position, and to splice the obtained spatial position encoding of each image with the corresponding image block embedding vector to obtain the spatial image block embedding vector of each image; Add a module for forming a spatial image block embedding sequence with all spatial image block embedding vectors, and adding a learnable classification tag at the front end of the spatial image block embedding sequence to obtain a new spatial image block embedding sequence; The fusion module is used to input the new spatial image patch embedding sequence into the Transformer model, extract the global information in all spatial image patch embedding vectors based on the proxy attention mechanism, update the learnable classification label, and obtain the fused classification label; The prediction module is used to use the fused classification labels to predict the target category through a multi-layer perceptron.

7. The multi-view target image fusion device according to claim 6, characterized in that: The spatial position encoding of the azimuth, pitch angle and distance of each image relative to the shooting position is used for: Normalize the azimuth, elevation, and distance of each image relative to the shooting position to obtain the normalized spatial position vector of each image; The normalized spatial position vector of each image is transformed through a linear mapping layer to obtain the spatial position encoding of each image.

8. The multi-view target image fusion device according to claim 6, wherein: The fusion module is used to: The new spatial image patch embedding sequence is input into the Transformer model, and the learnable classification label and each spatial image patch embedding vector are mapped into the corresponding query vector, key vector and value vector respectively through the learnable linear transformation of the Transformer model; Calculate the proxy score for each query vector through the perceptron; Select a preset number of query vectors with the highest proxy scores from all query vectors as proxy vectors; Taking each proxy vector as a query, perform attention calculation with each key vector and value vector respectively to obtain the converged features of each proxy vector; Taking the query vector corresponding to the learnable classification label as the query, perform attention calculation with the aggregated features of each proxy vector, and update the value vector corresponding to the learnable classification label; The value vector corresponding to the updated learnable classification label is used as the fused classification label.

9. A multi-view target image fusion device, characterized in that: The multi-perspective target image fusion device includes a processor, a memory, and a multi-perspective target image fusion program stored in the memory and executable by the processor, wherein when the multi-perspective target image fusion program is executed by the processor, the steps of the multi-perspective target image fusion method as described in any one of claims 1 to 5 are implemented.

10. A readable storage medium, characterized in that: The readable storage medium stores a multi-view target image fusion program, wherein when the multi-view target image fusion program is executed by a processor, the steps of the multi-view target image fusion method according to any one of claims 1 to 5 are implemented.

Images