A target matching method based on cross-domain multimodal fusion coding

Abstract

The present application relates to a target matching method based on cross-domain multimodal fusion coding, which is based on multi-view image data, lidar point cloud data and keyword text data to be detected, and obtains visual descriptors, viewpoint descriptors and second keyword feature vectors respectively; fuses the visual descriptor and viewpoint descriptor into a multi-view fusion 3D expression feature vector; linearly transforms the second keyword feature vector to obtain a third keyword feature vector; and splices the multi-view fusion 3D expression feature vector and the third keyword feature vector to obtain a first joint expression vector; passes the first joint expression vector through three layers of transformer coding blocks to output a three-level joint expression vector; and finds the part with the highest cosine similarity between the three-level joint expression vector and the first keyword feature vector as the target matching result. This method effectively realizes the deep feature fusion of the three modalities of image, point cloud and text.

Description

Technical Field

[0001] The present application relates to the technical field of multi-modal information fusion of multiple unmanned equipment, and in particular to a target matching method based on cross-domain multimodal fusion coding. Background Art

[0002] Achieving accurate target matching is crucial in cross-domain, multi-unmanned vehicle application scenarios. However, this currently faces numerous challenges. Firstly, the heterogeneity of multimodal data (such as multi-view image data, LiDAR point cloud data, and text data) makes it difficult for traditional methods to effectively fuse their deep features. Secondly, the significant differences in perspectives when collecting data across multiple unmanned vehicles across multiple domains make traditional modal alignment methods inadequate. Furthermore, while the MLIP multimodal fusion coding target matching model, based on the improved GLIP model, has high performance, its training requires a large amount of data corresponding to the three modalities of image, point cloud, and text. Currently, very little such data is publicly available, severely restricting model training and application. Summary of the Invention

[0003] Based on this, it is necessary to provide a target matching method based on cross-domain multimodal fusion coding, which includes:

[0004] S1: Obtain keyword text data to be detected, multi-view image data of multiple unmanned equipment, and lidar point cloud data; split the image feature matrix obtained by encoding the multi-view image data to obtain visual descriptors; perform visual point screening on the point-by-point feature vector obtained by encoding the lidar point cloud data to obtain viewpoint descriptors; convert the first keyword feature vector extracted from the keyword text data into digital form to obtain a second keyword feature vector;

[0005] S2: Fusing the visual descriptor and the viewpoint descriptor into a multi-view fusion 3D expression feature vector; performing a linear transformation on the second keyword feature vector to obtain a third keyword feature vector; and concatenating the multi-view fusion 3D expression feature vector and the third keyword feature vector to obtain a first joint expression vector;

[0006] S3: Arrange three layers of transformer coding blocks. The input of the first layer of transformer coding blocks is the first joint expression vector. The input of the remaining two layers of transformer coding blocks is the concatenation of the output of the previous layer and the third keyword feature vector. The third layer of transformer coding blocks outputs the three-level joint expression vector.

[0007] S4: Find the part with the highest cosine similarity between the three-level joint expression vector and the first keyword feature vector as the target matching result.

[0008] Preferably, encoding the multi-view image data includes: passing the image data of each view through a convolutional neural network respectively to obtain an image feature matrix corresponding to each view.

[0009] Preferably, splitting the image feature matrix includes: performing slicing or pooling operations on the image feature matrix corresponding to any view in a specific dimension to obtain multiple visual descriptors corresponding to the view.

[0010] Preferably, encoding the lidar point cloud data includes: encoding the lidar point cloud data through PointNet to obtain a point-by-point feature vector for each point.

[0011] Preferably, screening the visible points of the point-by-point feature vectors includes:

[0012] Calculate the midpoint of the lidar point cloud data i The Euclidean distance between it and any other point, and multiply the Euclidean distance by the weight matrix to get the first product;

[0013] Traverse the remaining points in the lidar point cloud data and calculate the points i The first products corresponding to all other points and summing up all the calculated first products;

[0014] The summation result is passed through the activation function to get the point i Visibility score;

[0015] Point i The visibility score of the point is compared with the set threshold. i When the visibility score is greater than or equal to the set threshold, the point i The point-by-point feature vector of is used as a viewpoint descriptor; otherwise, it is discarded;

[0016] It traverses all points in the lidar point cloud data, compares the visibility scores of all points with the set threshold, and obtains multiple viewpoint descriptors.

[0017] Preferably, converting the first keyword feature vector extracted from the keyword text data into a digital form comprises:

[0018] Pass the keyword text data through the word vector model to obtain the first keyword feature vector;

[0019] The first keyword feature vector is split into a sequence of numbers to obtain a second keyword feature vector.

[0020] Preferably, fusing the visual descriptor and the viewpoint descriptor into a multi-view fused 3D expression feature vector comprises:

[0021] Sum the products of all visual descriptors in any view and the first fusion weight to obtain a second summation result;

[0022] Sum the second summation results under all views to obtain the sum of visual descriptors;

[0023] Sum the products of all viewpoint descriptors and the second fusion weight to obtain the sum of viewpoint descriptors;

[0024] The sum of visual descriptors and viewpoint descriptors is added together to obtain the multi-view fusion 3D expression feature vector.

[0025] Preferably, the step of concatenating the multi-view fusion 3D expression feature vector and the third keyword feature vector comprises:

[0026] Multiply the multi-view fusion 3D expression feature vector by the feature vector weight to obtain a second product;

[0027] The second product is added to the third keyword feature vector to obtain a first joint expression vector.

[0028] Preferably, the concatenation of the output of the first / second layer transformer encoding block and the third keyword feature vector includes:

[0029] Multiply the output of the first / second layer transformer encoding block by the feature vector weight to obtain the third product / fourth product;

[0030] Add the third product / fourth product to the third keyword feature vector to obtain the input of the second / third layer transformer encoding block.

[0031] Preferably, S4 includes:

[0032] Splitting the three-level joint expression vector into multiple first feature blocks;

[0033] Splitting the first keyword feature vector into a plurality of second feature blocks;

[0034] Calculate the cosine similarity between any first feature block and each second feature block to obtain a row / column of similarity values;

[0035] Traversing all first feature blocks, combining a row / column of similarity values ​​calculated based on each first feature block into a similarity matrix;

[0036] The first feature block having the highest similarity value with the second feature block in the similarity matrix is ​​used as the target matching result.

[0037] Beneficial effects: This method obtains visual descriptors, viewpoint descriptors, and a second keyword feature vector based on multi-view image data, lidar point cloud data, and keyword text data to be detected; fuses the visual descriptor and viewpoint descriptor into a multi-view fusion 3D expression feature vector; linearly transforms the second keyword feature vector to obtain a third keyword feature vector; and concatenates the multi-view fusion 3D expression feature vector and the third keyword feature vector to obtain a first joint expression vector; passes the first joint expression vector through a three-layer transformer encoding block to output a three-level joint expression vector; and finds the part with the highest cosine similarity between the three-level joint expression vector and the first keyword feature vector as the target matching result. This method adopts a keyword-cued multimodal fusion encoding method, which effectively realizes the deep feature fusion of the three modalities of image, point cloud, and text, solves the problem of heterogeneity of multimodal data, and thus improves the accuracy and reliability of target matching. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0039] Figure 1 This is a flowchart of a target matching method based on cross-domain multimodal fusion coding in an embodiment of the present application. DETAILED DESCRIPTION

[0040] To make the above-mentioned objects, features, and advantages of the present application more clearly understood, the specific embodiments of the present application are described in detail below with reference to the accompanying drawings. The following description sets forth many specific details to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar improvements without violating the scope of the present application. Therefore, the present application is not limited to the specific embodiments disclosed below.

[0041] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of such features. Throughout the description of this application, "plurality" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0042] like Figure 1As shown, this embodiment provides a target matching method based on cross-domain multimodal fusion coding, the method comprising:

[0043] S1: Obtain keyword text data to be detected, multi-view image data of multiple unmanned equipment, and lidar point cloud data; split the image feature matrix obtained by encoding the multi-view image data to obtain a visual descriptor; perform visual point screening on the point-by-point feature vector obtained by encoding the lidar point cloud data to obtain a viewpoint descriptor; convert the first keyword feature vector extracted from the keyword text data into a digital form to obtain a second keyword feature vector.

[0044] Multi-view image data contains image information obtained from different perspectives and can provide rich appearance features; lidar point cloud data can accurately describe the spatial structure of the target; and keyword text data provides key clues for target matching from a semantic level.

[0045] In this embodiment, splitting the image feature matrix obtained by encoding the multi-view image data includes:

[0046] The image data of each view is passed through the convolutional neural network to obtain the image feature matrix corresponding to each view;

[0047] The image feature matrix corresponding to any view is sliced ​​or pooled in a specific dimension to obtain multiple visual descriptors corresponding to the view. The formula is expressed as:

[0048] ;

[0049] in, Indicates the v The first view j visual descriptors; s Indicates the stride of slicing or pooling; Indicates the v The image feature matrix corresponding to each view; x Represents the image feature matrix Index in the height dimension; each visual descriptor corresponds to a local window, and the height range of the window is ( x The value range of , that is, j The windows cover the height of the continuous s OK; y Represents the image feature matrix Indexes along the width dimension, y The value range is , that is, traverse the entire width of the image feature matrix; Represents the image feature matrix Perform full slicing operation on the channel dimension, that is, extract the spatial position All channel characteristics of ; n Represents the image feature matrix The width dimension is the total number of elements in the image feature matrix in the width direction.

[0050] The meaning of this formula is:

[0051] For the first Image feature matrix of each view (Dimensions are , i.e. height × width × channel), in the In a local window (height range , width range ), perform pooling operations (average pooling) along the height and width dimensions, and retain all information in the channel dimension, and finally generate the first The first view visual descriptors .

[0052] Pooling stride Determines the height range of the window, each window covers The final number of visual descriptors is .

[0053] Example: Assuming the image feature matrix The size is (i.e. height = 8, width = 10, channels = 256), pooling stride ,but:

[0054] The height range of each window is Line (such as correspond , correspond , and so on), generating Visual descriptors ( ).

[0055] For each window The spatial region performs channel-dimensional feature averaging, and finally each visual descriptor The dimension is (Same as the number of channels).

[0056] This operation realizes the extraction of feature expressions of local regions from multi-view images, providing fine-grained visual information for subsequent cross-modal fusion.

[0057] Furthermore, encoding the lidar point cloud data includes: encoding the lidar point cloud data through PointNet to obtain a point-by-point feature vector of each point.

[0058] Furthermore, the visual point screening of the point-by-point feature vector includes:

[0059] Calculate the midpoint of the lidar point cloud data i The Euclidean distance between it and any other point, and multiply the Euclidean distance by the weight matrix to get the first product;

[0060] Traverse the remaining points in the lidar point cloud data and calculate the points i The first products corresponding to all other points and summing up all the calculated first products;

[0061] The summation result is passed through the activation function to get the point i Visibility score;

[0062] Point i The visibility score of the point is compared with the set threshold. i When the visibility score is greater than or equal to the set threshold, the point i The point-by-point feature vector of is used as a viewpoint descriptor; otherwise, it is discarded;

[0063] It traverses all points in the lidar point cloud data, compares the visibility scores of all points with the set threshold, and obtains multiple viewpoint descriptors.

[0064] In this embodiment, converting the first keyword feature vector extracted from the keyword text data into a digital form includes:

[0065] Pass the keyword text data through the word vector model to obtain the first keyword feature vector;

[0066] The first keyword feature vector is split into a sequence of numbers to obtain a second keyword feature vector.

[0067] This embodiment also provides a KGM keyword generation pre-training module, which needs to be pre-trained:

[0068] 1. Data input: Input multi-view image data , LiDAR point cloud data , text data , these data contain different modal information of the target.

[0069] 2. Feature extraction:

[0070] Multi-view image data For feature extraction, convolutional neural network is used , as described above, we get the image feature vector .

[0071] LiDAR point cloud data Perform feature extraction and use PointNet to obtain point cloud feature vectors , as described above in the PointNet encoding process.

[0072] For text data Perform feature extraction and use the word vector model to obtain the text feature vector , as mentioned above.

[0073] 3. Point cloud modal text information acquisition: point cloud feature vector is the weight, text feature vector For the value, enter Layer transformer encoding block is encoded. Assume that The input of the layer transformer encoding block is , the output is ,but:

[0074] ;

[0075] in, It is a multi-head attention mechanism. is a feedforward neural network. Layer encoding to obtain the text information of interest in the point cloud modality .

[0076] 4. Image modal text information acquisition: image feature vector is the weight, text feature vector For the value, enter The layer transformer encoding block is encoded, similar to the acquisition of text information in the point cloud modality, to obtain the text information of interest in the image modality .

[0077] 5. Keyword text data generation: the text information of the point cloud modality and text information that the image modality focuses on To merge, for example The fused information is then fed into Layer transformer decoding block, assuming the The input of the layer transformer decoding block is , the output is ,but:

[0078] ;

[0079] in, is the decoding layer operation, Is the output of the encoding phase. Finally, the generated keyword text data is obtained These generated keyword text data serve as an input to the model, providing key semantic information for model training.

[0080] 6. Adaptive layer adjustment: This module sets up an adaptive layer adjustment module consisting of an input layer, an output layer, and a fully connected layer. Assume that the correlation measure between image data and text data is , the correlation measure between lidar point cloud data and text data is , calculate the adjusted number of layers through the fully connected layer :

[0081] ;

[0082] in 、 are the first and second weight matrices of the fully connected layer, is the first bias term of the fully connected layer, is the sigmoid activation function, This module dynamically adjusts the number of layers of the transformer encoding block and decoding block according to the strength of the correlation by learning the correlation between image data and text data, and between lidar point cloud data and text data. When the correlation is high, the number of layers can be appropriately reduced to improve efficiency; when the correlation is low, the number of layers can be increased to better mine potential information, thereby optimizing the effect of keyword generation.

[0083] After training is complete, only multi-view image data is input , LiDAR point cloud data , after dynamically adjusting the number of layers The KGM keyword generation module can generate keywords that match the input data. These keywords are used for pre-training of the target matching model, effectively solving the problem of insufficient training data.

[0084] Furthermore, an adaptive layer adjustment module can be implemented within the KGM keyword generation pre-training module. This module dynamically adjusts the number of layers in the transformer encoding and decoding blocks based on the correlation between image and text data, and between LiDAR point cloud data and text data. This optimization step aims to improve keyword generation while reducing unnecessary computational overhead. When correlation is high, the number of layers is appropriately reduced to improve efficiency; when correlation is low, the number of layers is increased to better mine potential information.

[0085] This module is integrated into the Transformer encoding block / decoding block stacking architecture and mainly functions in the cross-modal data interaction stage.

[0086] The specific implementation is as follows:

[0087] 1. Input feature correlation calculation layer,

[0088] Position: The cross-modal interaction layer located before the multimodal feature input transformer encoding block.

[0089] Function: Generates a dynamic correlation coefficient (ranging from 0.6 to 1.2) by calculating the cosine similarity matrix between image and text, and point cloud and text, to quantify the semantic association strength between image and text, and point cloud and text.

[0090] Calculate the cross-attention score for image and text features and normalize it to the correlation coefficient;

[0091] The lidar point cloud and text features generate weighted similarity through a gating mechanism.

[0092] 2. Dynamic layer controller,

[0093] Position: Embedded between residual connections in a stack of Transformer encoder blocks.

[0094] The workflow includes:

[0095] Correlation threshold judgment:

[0096] When the correlation coefficient is greater than 0.9, the layer compression mode is activated, retaining only one layer of coding blocks (Layer 0) and one layer of residual layers (Layer 1), and skipping the calculation of the subsequent four layers;

[0097] When the correlation coefficient is < 0.8, the full 6-layer encoding block is enabled, and an additional feature decoupling layer is inserted after Layer 3;

[0098] Otherwise, activate layer compression mode, retaining only three coding layers and one residual layer.

[0099] The residual connection path is dynamically selected through a learnable gating network.

[0100] Residual gating formula:

[0101] ;

[0102] in, Indicates the number of layers actually enabled by the coding block (according to the correlation coefficient Dynamic adjustment); Represents a binary gating function (valued at 0 or 1), controlling the Whether the layer participates in the calculation; : No. The feedforward calculation module of the layer Transformer encoding block; Indicates the The hidden state of the layer encoding block.

[0103] 3. Adaptive linkage of decoding blocks,

[0104] Position: Synchronously adjusted in the crisscross attention module at the decoding block layer.

[0105] Implementation:

[0106] The correlation coefficients output by the encoding block are passed to the decoding block via a shared weight matrix.

[0107] The decoding block dynamically adjusts based on the received coefficients:

[0108] When the correlation is high (correlation coefficient > 0.9), only one layer of decoding blocks is retained for feature mapping;

[0109] When the correlation is low (correlation coefficient < 0.8), the 3-layer decoding block is enabled to enhance semantic reconstruction;‌

[0110] Otherwise, the layer 2 decoding block is enabled.

[0111] Through the above-mentioned adaptive layer adjustment process, the performance and efficiency of the model can be further improved.

[0112] S2: Fuse the visual descriptor and the viewpoint descriptor into a multi-view fusion 3D expression feature vector; linearly transform the second keyword feature vector to obtain a third keyword feature vector; and concatenate the multi-view fusion 3D expression feature vector and the third keyword feature vector to obtain a first joint expression vector.

[0113] Specifically, fusing the visual descriptor and the viewpoint descriptor into a multi-view fused 3D expression feature vector includes:

[0114] Sum the products of all visual descriptors in any view and the first fusion weight to obtain a second summation result;

[0115] Sum the second summation results under all views to obtain the sum of visual descriptors;

[0116] Sum the products of all viewpoint descriptors and the second fusion weight to obtain the sum of viewpoint descriptors;

[0117] The sum of visual descriptors and viewpoint descriptors is added together to obtain the multi-view fusion 3D expression feature vector.

[0118] Furthermore, the second keyword feature vector is subjected to a linear transformation including:

[0119] Multiply the second key eigenvector by the linear weight matrix to obtain a fifth product;

[0120] The fifth product is added to the bias term to obtain the third keyword feature vector.

[0121] Furthermore, the step of concatenating the multi-view fusion 3D expression feature vector and the third keyword feature vector includes:

[0122] Multiply the multi-view fusion 3D expression feature vector by the feature vector weight to obtain a second product;

[0123] The second product is added to the third keyword feature vector to obtain a first joint expression vector.

[0124] In this embodiment, the value of the feature vector weight is 3.

[0125] This can ensure the dominant position of the multi-view fusion 3D expression feature vector in the fusion process and give full play to its main description role of the target features.

[0126] S3: Arrange three layers of transformer coding blocks. The input of the first layer of transformer coding blocks is the first joint expression vector. The input of the remaining two layers of transformer coding blocks is the concatenation of the output of the previous layer and the third keyword feature vector. The third layer of transformer coding blocks outputs the three-level joint expression vector.

[0127] Specifically, the vectors input into the encoding block are further integrated and optimized through the multi-head attention mechanism in the transformer encoding block. The feature information is further mined and fused through three layers of transformer encoding blocks, gradually enhancing the fusion effect of multimodal features.

[0128] Furthermore, the concatenation of the output of the first / second layer transformer encoding block and the third keyword feature vector includes:

[0129] Multiply the output of the first / second layer transformer encoding block by the feature vector weight to obtain the third product / fourth product;

[0130] Add the third product / fourth product to the third keyword feature vector to obtain the input of the second / third layer transformer encoding block.

[0131] The advantages of three-step concatenation and transformer encoding are:

[0132] Gradually strengthen multimodal feature fusion: During the three-fold concatenation and encoding process, starting from the second layer, the keyword feature vector is weightedly concatenated with the output of the previous layer's transformer encoding block. This information is further integrated and optimized through the transformer encoding block. This process gradually strengthens the fusion of multimodal features, allowing the resulting joint expression vector to more comprehensively and accurately reflect the target's multimodal characteristics.

[0133] Improved target matching accuracy and robustness: Through multiple concatenation and encoding steps, the model learns richer feature representations, enabling it to more accurately identify the most similar feature vectors to the keyword being detected during target matching. This helps improve target matching accuracy and robustness, especially in cross-domain, multi-unmanned equipment scenarios, where significant differences in viewpoints and high data heterogeneity are present, ensuring good matching results.

[0134] Reduced training time: Although the three-loop concatenation increases the amount of computation, thanks to the efficient processing capabilities of the transformer encoding block and the weight setting of the keyword feature vectors during the concatenation process (the weight is relatively small), overall, this method reduces unnecessary computational overhead while ensuring the fusion effect, thereby reducing training time to a certain extent.

[0135] S4: Find the part with the highest cosine similarity between the three-level joint expression vector and the first keyword feature vector as the target matching result.

[0136] Specifically, this step includes:

[0137] Splitting the three-level joint expression vector into multiple first feature blocks;

[0138] Splitting the first keyword feature vector into a plurality of second feature blocks;

[0139] Calculate the cosine similarity between any first feature block and each second feature block to obtain a row / column of similarity values;

[0140] Traversing all first feature blocks, combining a row / column of similarity values ​​calculated based on each first feature block into a similarity matrix;

[0141] The first feature block having the highest similarity value with the second feature block in the similarity matrix is ​​used as the target matching result.

[0142] After the model training is completed, the feature vector is processed by the trained three-layer transformer encoding block and then matched with the target to be identified to obtain the final target matching result, which greatly improves the convenience and efficiency of the application.

[0143] The target matching method based on cross-domain multimodal fusion coding provided in this embodiment has the following beneficial effects:

[0144] 1. The proposed KGM keyword generation pre-training module can generate corresponding keyword text data using only image and point cloud data through training, which is used to train the target matching model. This module solves the problem of insufficient data required for training the MLIP multimodal fusion encoding target matching model, ensures the high generalization ability of the model, and enables the addition of targets to be identified at any time without retraining.

[0145] 2. The multimodal fusion encoding method based on keyword prompts effectively realizes the deep feature fusion of three modalities: image, point cloud, and text, while reducing training time and solving the problem of multimodal data heterogeneity.

[0146] 3. By fusing visual descriptors and viewpoint descriptors to generate a multi-view fusion 3D expression feature vector, it can handle the problem of huge differences in perspectives and cross-domain when collecting data from multiple unmanned equipment across domains, thereby improving the accuracy and reliability of target matching.

[0147] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0148] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A target matching method based on cross-domain multimodal fusion coding, characterized in that: include: S1: Obtain keyword text data to be detected, multi-view image data of multiple unmanned equipment, and lidar point cloud data; The image feature matrix obtained by encoding the multi-view image data is split to obtain a visual descriptor; the point-by-point feature vector obtained by encoding the lidar point cloud data is subjected to visual point screening to obtain a viewpoint descriptor; the first keyword feature vector extracted from the keyword text data is converted into a digital form to obtain a second keyword feature vector; S2: Fusing the visual descriptor and the viewpoint descriptor into a multi-view fusion 3D expression feature vector; performing a linear transformation on the second keyword feature vector to obtain a third keyword feature vector; and concatenating the multi-view fusion 3D expression feature vector and the third keyword feature vector to obtain a first joint expression vector; S3: Arrange three layers of transformer coding blocks. The input of the first layer of transformer coding blocks is the first joint expression vector. The input of the remaining two layers of transformer coding blocks is the concatenation of the output of the previous layer and the third keyword feature vector. The third layer of transformer coding blocks outputs the three-level joint expression vector. S4: Find the part with the highest cosine similarity between the three-level joint expression vector and the first keyword feature vector as the target matching result.

2. The target matching method based on cross-domain multimodal fusion coding according to claim 1 is characterized in that Encoding multi-view image data includes: passing the image data of each view through a convolutional neural network to obtain an image feature matrix corresponding to each view.

3. The target matching method based on cross-domain multimodal fusion coding according to claim 1 is characterized in that Splitting the image feature matrix includes: performing slicing or pooling operations on the image feature matrix corresponding to any view in a specific dimension to obtain multiple visual descriptors corresponding to the view, where the specific dimension includes a height and a width dimension.

4. The target matching method based on cross-domain multimodal fusion coding according to claim 1 is characterized in that Encoding the lidar point cloud data includes: encoding the lidar point cloud data through PointNet to obtain a point-by-point feature vector for each point.

5. The target matching method based on cross-domain multimodal fusion coding according to claim 1 is characterized in that: Visual point screening of point-by-point feature vectors includes: Calculate the midpoint of the lidar point cloud data i The Euclidean distance between it and any other point, and multiply the Euclidean distance by the weight matrix to get the first product; Traverse the remaining points in the lidar point cloud data and calculate the points i The first products corresponding to all other points and summing up all the calculated first products; The summation result is passed through the activation function to get the point i Visibility score; Point i The visibility score of the point is compared with the set threshold. i When the visibility score is greater than or equal to the set threshold, the point i The point-by-point feature vector of is used as a viewpoint descriptor; otherwise, it is discarded; It traverses all points in the lidar point cloud data, compares the visibility scores of all points with the set threshold, and obtains multiple viewpoint descriptors.

6. The target matching method based on cross-domain multimodal fusion coding according to claim 1 is characterized in that The step of converting the first keyword feature vector extracted from the keyword text data into a digital form includes: Pass the keyword text data through the word vector model to obtain the first keyword feature vector; The first keyword feature vector is split into a sequence of numbers to obtain a second keyword feature vector.

7. The target matching method based on cross-domain multimodal fusion coding according to claim 1 is characterized in that: The fusing of the visual descriptor and the viewpoint descriptor into a multi-view fused 3D expression feature vector comprises: Sum the products of all visual descriptors in any view and the first fusion weight to obtain a second summation result; Sum the second summation results under all views to obtain the sum of visual descriptors; Sum the products of all viewpoint descriptors and the second fusion weight to obtain the sum of viewpoint descriptors; The sum of visual descriptors and viewpoint descriptors is added together to obtain the multi-view fusion 3D expression feature vector.

8. The target matching method based on cross-domain multimodal fusion coding according to claim 1 is characterized in that: The step of concatenating the multi-view fusion 3D expression feature vector and the third keyword feature vector includes: Multiply the multi-view fusion 3D expression feature vector by the feature vector weight to obtain a second product; The second product is added to the third keyword feature vector to obtain a first joint expression vector.

9. The target matching method based on cross-domain multimodal fusion coding according to claim 1 is characterized in that: The concatenation of the output of the first / second layer transformer encoding block and the third keyword feature vector includes: Multiply the output of the first / second layer transformer encoding block by the feature vector weight to obtain the third product / fourth product; Add the third product / fourth product to the third keyword feature vector to obtain the input of the second / third layer transformer encoding block.

10. The target matching method based on cross-domain multimodal fusion coding according to claim 1, characterized in that S4 include: Splitting the three-level joint expression vector into multiple first feature blocks; Splitting the first keyword feature vector into a plurality of second feature blocks; Calculate the cosine similarity between any first feature block and each second feature block to obtain a row / column of similarity values; Traversing all first feature blocks, combining a row / column of similarity values ​​calculated based on each first feature block into a similarity matrix; The first feature block having the highest similarity value with the second feature block in the similarity matrix is ​​used as the target matching result.

Images