Cross-modal adaptive feature extraction method for target detection

Abstract

The embodiment of the invention provides a cross-modal adaptive feature extraction method for target detection, and the method comprises the steps: constructing an initial multi-modal feature representation based on various modal data; performing preliminary fusion processing on the multi-modal feature representation; adjusting the weight of each modal feature through a cross-modal adaptive layer; inputting the weighted features into a detection model for target detection; and parameters of the cross-modal adaptive layer are optimized according to detection result feedback. Through the scheme of the embodiment of the invention, the problem that alignment is difficult due to feature representation difference under the condition that cross-modal data pairing is inconsistent can be solved; the problem of model deviation caused by data deviation under the condition that feature distribution among different modes is unbalanced is solved; the problem of low calculation efficiency caused by too high feature dimension in a multi-modal feature fusion process is solved; and the problem of feature information loss caused by poor quality of partial modal data during feature extraction is solved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a cross-modal adaptive feature extraction method for target detection. Background Art

[0002] With the development of computer vision technology, target detection plays an increasingly important role in many application fields, such as intelligent transportation systems, security monitoring, and medical diagnosis. Traditional single-modal target detection methods have achieved good results in specific scenarios, but when faced with complex and changing application environments, single-modal methods often have certain limitations. In order to overcome these limitations, researchers have begun to explore methods for target detection using cross-modal data in recent years.

[0003] Cross-modal object detection methods aim to improve the accuracy and robustness of detection by fusing data from multiple modalities (such as images, videos, audio, text, etc.) and utilizing complementary information between different modalities. In theory, this method can make up for the shortcomings of single-modal data in some aspects, thereby achieving more stable and reliable object detection. However, in practical applications, cross-modal object detection faces a series of challenges:

[0004] 1. Difficulty of feature alignment: Since data from different modalities usually have different feature representations, how to effectively align these features becomes a difficult problem. Especially when the modal data pairing is inconsistent, the accuracy of feature alignment directly affects the detection effect.

[0005] 2. Unbalanced feature distribution: Data of different modalities may have large differences in feature distribution, which may lead to data bias in the fusion process, causing the model to tend to rely on information from certain modalities while ignoring important information from other modalities.

[0006] 3. High-dimensional feature processing: When fusing multimodal features, the demand for computing resources increases due to the high feature dimension, which not only increases the computing cost but also may reduce the real-time performance of the algorithm.

[0007] 4. Data quality issues: In actual application scenarios, the data quality of some modalities may not be as good as expected. For example, due to noise interference caused by equipment failure or environmental factors, these problems may lead to the loss of important feature information.

[0008] 5. Overfitting risk: During the training process, noise interference between modalities may cause the model to learn incorrect feature associations, increasing the risk of overfitting and thus affecting the model's ability to generalize to new data.

[0009] In view of the above problems, developing an effective cross-modal adaptive feature extraction method to solve the problems of feature alignment, uneven feature distribution, low computational efficiency and data quality, and reduce the risk of overfitting has become a key technical problem that needs to be solved urgently. The purpose of the present invention is to provide a cross-modal adaptive feature extraction method that can effectively cope with the above challenges to improve the performance of target detection. Summary of the invention

[0010] In view of the problems existing in the above-mentioned existing methods for optimizing the network retrieval capability of large models, the present invention is proposed.

[0011] Therefore, the problems to be solved by the present invention are: first, since data of different modalities usually have different feature representations, how to effectively align these features becomes a difficult problem, especially when the modal data pairing is inconsistent, the accuracy of feature alignment directly affects the detection effect; second, data of different modalities may have large differences in feature distribution, which may lead to data bias in the fusion process, and then make the model tend to rely on the information of certain modalities and ignore the important information of other modalities; third, when fusing multimodal features, due to the high feature dimension, the demand for computing resources also increases, which not only increases the computing cost, but also may reduce the real-time performance of the algorithm; fourth, in actual application scenarios, the data quality of certain modalities may not be as expected, such as due to noise interference caused by equipment failure or environmental factors, these problems may lead to the loss of important feature information; fifth, during the training process, noise interference between modalities may cause the model to learn incorrect feature associations, increase the risk of overfitting, and thus affect the generalization ability of the model on new data. These five aspects are thorny problems that need to be considered and solved when using large pre-trained models for information retrieval optimization in a networked environment.

[0012] In order to solve the above technical problems, the present invention provides the following technical solutions: a cross-modal adaptive feature extraction method for target detection, comprising the following steps:

[0013] Based on multi-modal data, construct initial multi-modal feature representation;

[0014] Performing preliminary fusion processing on the multimodal feature representation;

[0015] Through the cross-modal adaptation layer, the weights of each modality feature are adjusted;

[0016] Input the weighted features into the detection model for target detection;

[0017] According to the detection result feedback, the parameters of the cross-modal adaptation layer are optimized.

[0018] As a preferred solution of the method for optimizing the large model network retrieval capability of the present invention, the step of constructing an initial multimodal feature representation based on multiple modal data includes:

[0019] Collect data from at least two different modalities, such as imagery and radar data;

[0020] Preprocess the data of each modality, including data normalization and noise filtering;

[0021] Based on the preprocessed data, extract the respective feature representation;

[0022] The feature representations of all modalities are preliminarily integrated to form an initial multimodal feature representation.

[0023] As a preferred solution of the method for optimizing the large model network retrieval capability of the present invention, the step of extracting respective feature representations based on the preprocessed data includes:

[0024] Perform convolutional neural network feature extraction on image data to obtain image feature vectors;

[0025] Perform Fourier transform on radar data to obtain radar spectrum characteristics;

[0026] Based on the eigenvectors of the two modal data, the correlation of the eigenvectors is calculated;

[0027] When the correlation coefficient is greater than the threshold C, the two modal data are selected for subsequent processing, and the formula is R_{xy}>C, where R_{xy} represents the correlation coefficient between the image feature vector and the radar feature vector, and C is the preset threshold.

[0028] As a preferred solution of the method for optimizing the large model network retrieval capability of the present invention, the step of calculating the correlation of the feature vectors based on the feature vectors of the two modal data includes:

[0029] Calculate the dot product of the image feature vector and the radar feature vector;

[0030] Calculate the norm of the two eigenvectors;

[0031] Based on the dot product and norm, the correlation coefficient R_{xy}=frac{XcdotY}{|X||Y|} is calculated, where X and Y are the image feature vector and radar feature vector respectively;

[0032] When the correlation coefficient R_{xy} is greater than the threshold C, the two modal data are selected for subsequent processing, and the formula is R_{xy}>C, where R_{xy} represents the correlation coefficient between the image feature vector and the radar feature vector, and C is the preset threshold.

[0033] As a preferred solution of the method for optimizing the large model network retrieval capability of the present invention, the step of calculating the correlation coefficient R_{xy}=frac{XcdotY}{|X||Y|} based on the dot product and norm includes:

[0034] Calculate the inner product (XcdotY) of the image feature vector and the radar feature vector;

[0035] Calculate the Euclidean norm of the image feature vector (|X|);

[0036] Calculate the Euclidean norm (|Y|) of the radar feature vector;

[0037] Based on the inner product and the norm, the correlation coefficient R_{xy}=frac{XcdotY}{|X||Y|} is calculated.

[0038] As a preferred solution of the method for optimizing the large model network retrieval capability of the present invention, the steps based on the inner product (XcdotY) of the image feature vector and the radar feature vector include:

[0039] Multiply the corresponding components of the image feature vector and the radar feature vector;

[0040] Sum the products of all components;

[0041] When the inner product XcdotY is greater than the threshold T, the feature vector is considered to have a high similarity. The formula is XcdotY>T, where XcdotY represents the inner product and T is the preset threshold.

[0042] The inner product result is used for subsequent correlation calculation.

[0043] As a preferred solution of the method for optimizing the large model network retrieval capability of the present invention, the step based on summing the product results of all components includes:

[0044] Initialize a cumulative variable S to 0;

[0045] Traverse the corresponding components of the image feature vector and the radar feature vector, and calculate the product of each group of components;

[0046] Add the product of each group's weight to the cumulative variable S;

[0047] When the cumulative variable S is greater than the threshold T, the feature vector is considered to have a higher similarity, and the formula is S>T, where S represents the cumulative variable and T is the preset threshold.

[0048] The beneficial effects of the present invention are: constructing an initial multimodal feature representation based on multiple modal data; performing preliminary fusion processing on the multimodal feature representation; adjusting the weights of each modal feature through a cross-modal adaptive layer; inputting the weighted features into a detection model for target detection; and optimizing the parameters of the cross-modal adaptive layer based on the feedback of the detection results. The solution of the embodiment of the present disclosure can solve the following problems:

[0049] 1. The alignment difficulty caused by differences in feature representation when cross-modal data pairing is inconsistent.

[0050] 2. Model bias caused by data deviation when feature distribution between different modalities is unbalanced.

[0051] 3. The problem of low computational efficiency due to high feature dimensions in the process of multimodal feature fusion.

[0052] 4. The problem of feature information loss due to poor quality of some modal data during feature extraction.

[0053] 5. Overfitting problem caused by noise interference between modalities during training. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0055] Figure 1 Flowchart of a cross-modal adaptive feature extraction method for target detection. DETAILED DESCRIPTION

[0056] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings.

[0057] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0058] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor is it a separate or selective embodiment that is mutually exclusive with other embodiments.

[0059] Example 1

[0060] Reference Figure 1 , describing a cross-modal adaptive feature extraction method for target detection of the present invention, comprising the following steps:

[0061] Based on multi-modal data, construct initial multi-modal feature representation;

[0062] Performing preliminary fusion processing on the multimodal feature representation;

[0063] Through the cross-modal adaptation layer, the weights of each modality feature are adjusted;

[0064] Input the weighted features into the detection model for target detection;

[0065] According to the detection result feedback, the parameters of the cross-modal adaptation layer are optimized.

[0066] Furthermore, based on the multi-modal data, the steps of constructing an initial multi-modal feature representation include:

[0067] Collect data from at least two different modalities, such as imagery and radar data;

[0068] Preprocess the data of each modality, including data normalization and noise filtering;

[0069] Based on the preprocessed data, extract the respective feature representation;

[0070] Preliminarily integrate the feature representations of all modalities to form an initial multimodal feature representation;

[0071] Furthermore, based on the preprocessed data, the steps of extracting respective feature representations include:

[0072] Perform convolutional neural network feature extraction on image data to obtain image feature vectors;

[0073] Perform Fourier transform on radar data to obtain radar spectrum characteristics;

[0074] Based on the eigenvectors of the two modal data, the correlation of the eigenvectors is calculated;

[0075] When the correlation coefficient is greater than the threshold C, the two modal data are selected for subsequent processing, and the formula is R_{xy}>C, where R_{xy} represents the correlation coefficient between the image feature vector and the radar feature vector, and C is the preset threshold;

[0076] Furthermore, based on the eigenvectors of the two modal data, the step of calculating the correlation of the eigenvectors includes:

[0077] Calculate the dot product of the image feature vector and the radar feature vector;

[0078] Calculate the norm of the two eigenvectors;

[0079] Based on the dot product and norm, the correlation coefficient R_{xy}=frac{XcdotY}{|X||Y|} is calculated, where X and Y are the image feature vector and radar feature vector respectively;

[0080] When the correlation coefficient R_{xy} is greater than the threshold C, the two modal data are selected for subsequent processing, and the formula is R_{xy}>C, where R_{xy} represents the correlation coefficient between the image feature vector and the radar feature vector, and C is the preset threshold;

[0081] Further, based on the dot product and the norm, the step of calculating the correlation coefficient R_{xy}=frac{XcdotY}{|X||Y|} includes:

[0082] Calculate the inner product (XcdotY) of the image feature vector and the radar feature vector;

[0083] Calculate the Euclidean norm of the image feature vector (|X|);

[0084] Calculate the Euclidean norm (|Y|) of the radar feature vector;

[0085] Based on the inner product and norm, calculate the correlation coefficient R_{xy}=frac{XcdotY}{|X||Y|};

[0086] Further, the steps based on the inner product (XcdotY) of the image feature vector and the radar feature vector include:

[0087] Multiply the corresponding components of the image feature vector and the radar feature vector;

[0088] Sum the products of all components;

[0089] When the inner product XcdotY is greater than the threshold T, the feature vector is considered to have a high similarity. The formula is XcdotY>T, where XcdotY represents the inner product and T is the preset threshold.

[0090] The inner product result is used for subsequent correlation calculation;

[0091] Further, the step of summing the product results of all components includes:

[0092] Initialize a cumulative variable S to 0;

[0093] Traverse the corresponding components of the image feature vector and the radar feature vector, and calculate the product of each group of components;

[0094] Add the product of each group's weight to the cumulative variable S;

[0095] When the cumulative variable S is greater than the threshold T, the feature vector is considered to have a higher similarity, and the formula is S>T, where S represents the cumulative variable and T is the preset threshold.

[0096] Reference Figure 1 , describing a cross-modal adaptive feature extraction method for target detection of the present invention. The method mainly includes the following five steps: constructing an initial multi-modal feature representation based on multiple modal data; performing preliminary fusion processing on the multi-modal feature representation; adjusting the weights of each modal feature through a cross-modal adaptive layer; inputting the weighted features into a detection model for target detection; and optimizing the parameters of the cross-modal adaptive layer based on the detection result feedback.

[0097] Based on multi-modal data, an initial multimodal feature representation is constructed. This step aims to extract the original features of each modality from different types of perception data (such as images, audio, lidar point clouds, etc.). Different modalities provide information from different angles of the same object or environment, which is very beneficial for target recognition in complex scenes. Specifically, for example, in the vehicle recognition task in the intelligent transportation system, RGB images collected by the camera, thermal images obtained by the infrared camera, and three-dimensional point clouds generated by the lidar can be used as input sources. The information captured by each sensor has different representational characteristics. Images are suitable for extracting visual information such as shape and color. Point clouds can reflect the specific spatial structure and size of the target. Thermal images can be used to capture temperature information to assist in distinguishing organisms such as humans and animals. These heterogeneous information together make it possible to define targets more accurately.

[0098] The multimodal feature representation is subjected to preliminary fusion processing. Preliminary fusion is the process of combining feature vectors from different sources by means of simple arithmetic operations, feature-level splicing, or early fusion network architecture. This not only retains the unique expressions of each modality, but also enhances the ability to learn the multi-dimensional features of the target object. For example, in the above-mentioned vehicle recognition application scenario, the convolutional neural network can be trained separately to process the RGB image to obtain its feature map, the deep convolutional generative adversarial network (DC-GAN) model can be used to process the point cloud to obtain three-dimensional features, and the pre-trained convolutional network can be used to extract thermal imaging features. Then, the outputs of these three models can be combined by element-by-element addition or splicing to form a multimodal comprehensive representation vector.

[0099] Through the cross-modal adaptive layer, the weights of each modal feature are adjusted. This stage involves designing specific mechanisms to dynamically adjust the contribution of different modal features. This step is particularly important for problems such as inconsistent cross-modal data pairing, differences in feature representation, and unbalanced feature distribution. It allows the network to learn to focus on which modalities are most critical in the current context and which may contain redundant information or noise. For example, if in a specific urban monitoring environment, low light conditions at night seriously affect the camera imaging quality, the corresponding image feature weights should be reduced; if there are multiple high-precision radar devices installed on a certain section of road, the trust in this part of the data can be increased, that is, their corresponding weights can be increased.

[0100] The weighted features are input into the detection model for target detection. The target detection module here can use popular frameworks such as FasterR-CNN, YOLO or SSD, which can effectively combine the fusion features generated in the previous text to complete the instantiation detection task, that is, determine the existence and specific location of objects in the picture or video frame. In this step, the multimodal features that have been weighted by the adaptive mechanism will become the final decision basis, which will help improve the robustness and accuracy of object recognition in complex backgrounds and changing environments. Continuing with the example of urban traffic monitoring, extreme weather conditions may be encountered in actual deployment, such as rain, snow, fog and haze. At this time, the model needs to rely more on non-light input to ensure high performance even in harsh environments.

[0101] According to the feedback of the detection results, the parameters of the cross-modal adaptive layer are optimized. The last key step is to adjust the weight parameters in the adaptive layer based on the effect of target detection. This generally involves the back-propagation algorithm, the purpose of which is to enable the overall framework to automatically learn how to best combine and balance the information of each modality during continuous training. For example, after each training cycle, the system can evaluate the quality of all predictions relative to the ground truth labels (such as using precision and recall indicators), and pass the evaluation scores back to the adaptive layer mentioned above to update the importance score of each modality. In addition, in order to prevent the model from over-relying on a single data source and causing over-matching problems, regularization constraints or cross-validation mechanisms are introduced to suppress potential overfitting. Through such an iterative optimization process, the entire cross-modal target detection system gradually stabilizes and shows higher adaptability and generalization potential.

[0102] Next, the steps of constructing an initial multimodal feature representation based on multimodal data of the present invention are described.

[0103] Collect data from at least two different modalities, such as images and radar data. The key to this step is to ensure the diversity and reliability of data sources. For example, in an object detection scenario, high-quality image data can be obtained from a camera while radar point cloud data can be collected from a vehicle-mounted radar system. Together, these data provide information from different perspectives and dimensions, which contributes to the accuracy and comprehensiveness of subsequent feature extraction.

[0104] Preprocessing is performed on each modality data, including data normalization and noise filtering. The purpose of the preprocessing step is to remove or reduce the noise in the data to make it easier to process and analyze. Specifically, for image data, techniques such as histogram equalization and pixel normalization can be used, while for radar data, outliers can be removed through a Kalman filter. These preprocessing operations ensure the consistency and stability of the input data and improve the robustness of the model.

[0105] Based on the preprocessed data, the respective feature representations are extracted. Feature extraction is the process of performing in-depth analysis on the preprocessed data to extract its key information. For example, in image data, convolutional neural networks (CNNs) can be used to extract visual features such as edges and textures; in radar data, frequency domain features can be extracted through Fourier transform or point cloud distribution features can be extracted using clustering algorithms. The result of feature extraction is usually a multidimensional vector, where each dimension represents a specific attribute.

[0106] Perform a preliminary integration of the feature representations of all modalities to form an initial multimodal feature representation. The preliminary integration step is to merge the feature vectors of different modalities into a unified multimodal feature representation. For example, the features of each modality can be combined simply by concatenating the feature vectors of different modalities or by using a weighted average method. Let (mathbf{f}_1) be the image feature vector and (mathbf{f}_2) be the radar feature vector. The following formula can be used for concatenation: [

[0108] mathbf{F}=[mathbf{f}_1; mathbf{f}_2] ]

[0110] Here, ([mathbf{f}_1;mathbf{f}_2]) represents the vertical concatenation of two vectors, and (mathbf{F}) is the final initial multimodal feature representation. The concatenation operation is simple and effective, and can retain the original information of each modality, providing a basis for subsequent feature optimization and model training. In another embodiment, if the importance of each modality feature is different, a weighted average method can be used, for example: [

[0112] mathbf{F}=w_1mathbf{f}_1+w_2mathbf{f}_2 ]

[0114] Among them, (w_1) and (w_2) are the weight coefficients of image features and radar features respectively, satisfying (w_1+w_2=1), and (0leqw_1,w_2leq1). The optimal weight value can be determined through cross-validation and parameter adjustment to achieve the best feature integration effect.

[0115] The above steps form a rich initial multimodal feature representation through preprocessing, feature extraction and preliminary integration of multimodal data, laying the foundation for further target detection and cross-modal feature extraction.

[0116] Next, the steps of extracting respective feature representations based on the preprocessed data of the present invention are described. First, convolutional neural network feature extraction is performed on the image data to obtain an image feature vector. In this step, the convolutional neural network (CNN) extracts high-level abstract features from the image through multiple convolutional layers, pooling layers, and activation functions. Specifically, the convolutional layer captures local features in the image through different convolution kernels, the pooling layer reduces the spatial dimension of the feature map by downsampling, and activation functions such as ReLU increase the nonlinearity of the model. For example, an RGB image of size 224×224×3 is input, and after multiple layers of convolution operations, a 1024-dimensional feature vector is output.

[0117] Next, the radar data is Fourier transformed to obtain radar spectrum features. In this process, the radar signal in the time domain is converted into a frequency domain signal by fast Fourier transform (FFT), so as to extract the radar spectrum features. Specifically, assuming that the radar signal is a sequence of length 1024, FFT converts it into a frequency domain signal, and finally obtains a radar spectrum feature containing 1024 frequency components. In one embodiment, these frequency components reflect the intensity distribution of the radar signal at different frequencies.

[0118] Then, based on the feature vectors of the two modal data, the correlation of the feature vectors is calculated. In this step, the similarity between the image feature vector and the radar spectrum feature is measured by calculating the Pearson Correlation Coefficient. Specifically, assuming that the image feature vector is (mathbf{x}) and the radar feature vector is (mathbf{y}), the Pearson correlation coefficient (R_{xy}) is calculated as follows:

[0119] [R_{xy}=frac{sum_{i=1}^{N}(x_i-bar{x})(y_i-bar{y})}{sqrt{sum_{i=1}^{N}(x_i-bar{x})^2sum_{i=1}^{N}(y_i-bar{y})^2}}]

[0120] Where (N) is the dimension of the feature vector, (bar{x}) and (bar{y}) are the means of (mathbf{x}) and (mathbf{y}) respectively. The parameter (R_{xy}) ranges from -1 to 1, with values ​​closer to 1 indicating a more positive correlation, values ​​closer to -1 indicating a more negative correlation, and values ​​closer to 0 indicating no correlation. The optimal value is usually a high positive value, such as 0.8 or higher, indicating that the two feature vectors have a strong correlation.

[0121] Finally, when the correlation coefficient is greater than the threshold (C), the two modal data are selected for subsequent processing, and the formula is (R_{xy}>C). The threshold (C) here is used to screen feature combinations with higher correlation. Specifically, if the preset (C) is 0.7, only when (R_{xy}) is greater than 0.7, the image features and radar features can be further used for subsequent processing of target detection. This setting ensures that only those modal data that can provide effective information complementarity are selected, thereby improving the accuracy and robustness of target detection. For example, in an autonomous driving scenario, when the correlation between image data and radar data is high, it indicates that the two are highly complementary and can better jointly identify obstacles ahead.

[0122] Next, the steps of calculating the correlation of feature vectors based on the feature vectors of two modal data of the present invention are described. These steps include: calculating the dot product of the image feature vector and the radar feature vector; calculating the norm of the two feature vectors; calculating the correlation coefficient (R_{xy}=frac{XcdotY}{|X||Y|}) based on the dot product and the norm, where X and Y are the image feature vector and the radar feature vector respectively; when the correlation coefficient (R_{xy}) is greater than the threshold C, the two modal data are selected for subsequent processing, and the formula is (R_{xy}>C).

[0123] In the first step, the dot product of the image feature vector and the radar feature vector is calculated. The main purpose of this step is to measure the similarity between the two vectors through the dot product. The larger the dot product result of the image feature vector and the radar feature vector, the higher the similarity in features between the two. The formula for the dot product is (XcdotY=sum_{i=1}^{n}X_iY_i), where (X_i) and (Y_i) are the i-th elements of the image feature vector and the radar feature vector, respectively, and n is the dimension of the feature vector. For example, in one embodiment, assuming that the image feature vector is [2,4,6] and the radar feature vector is [3,3,3], the dot product result is 2*3+4*3+6*3=36.

[0124] The second step is to calculate the norm of the two feature vectors. The norm is used to measure the length or size of the vector, and the Euclidean norm (L2 norm) is usually used, and the formula is (|X|=sqrt{sum_{i=1}^{n}X_i^2}). The larger the norm, the stronger the strength or influence of the vector in the feature space. For example, in the same embodiment, the norm of the image feature vector [2,4,6] is (sqrt{2^2+4^2+6^2}=sqrt{56}approx7.48), and the norm of the radar feature vector [3,3,3] is (sqrt{3^2+3^2+3^2}=sqrt{27}approx5.20).

[0125] The third step is to calculate the correlation coefficient (R_{xy}=frac{XcdotY}{|X||Y|}) based on the dot product and norm. The correlation coefficient ranges from -1 to 1, where -1 indicates a complete negative correlation, 0 indicates no correlation, and 1 indicates a complete positive correlation. In the above embodiment,

[0126] (R_{xy}=frac{36}{7.48times5.20}approxfrac{36}{38.896}approx0.93). A high correlation coefficient indicates that the two feature vectors are highly similar in the feature space.

[0127] In the last step, when the correlation coefficient (R_{xy}) is greater than the threshold C, the two modal data are selected for subsequent processing, and the formula is (R_{xy}>C). The threshold C is a preset value used to filter data pairs with low correlation, thereby improving the effectiveness and accuracy of feature extraction. For example, in one embodiment, assuming that the preset threshold C is 0.8, then (R_{xy}=0.93>0.8), so the image feature vector and radar feature vector are selected for further target detection processing. The reason for this is that in practical applications, only selecting data with higher correlation can improve the performance and robustness of the model and reduce the interference of irrelevant data.

[0128] Next, the steps of calculating the correlation coefficient (R_{xy}=frac{XcdotY}{|X||Y|}) based on the dot product and norm of the present invention are described. First, the inner product (XcdotY) of the image feature vector (X) and the radar feature vector (Y) is calculated. Here, (X) and (Y) represent the feature vectors extracted from the image and the radar signal, respectively. The calculation formula of the inner product (XcdotY) is (XcdotY=sum_{i=1}^{n}x_iy_i), where (x_i) and (y_i) are the (i)th elements of the feature vectors (X) and (Y), respectively, and (n) is the dimension of the feature vector. The inner product result reflects the similarity of the two feature vectors in direction.

[0129] Next, the Euclidean norm (|X|) of the image feature vector is calculated. The calculation formula of the Euclidean norm is (|X|=sqrt{sum_{i=1}^{n}x_i^2}), which represents the length of the feature vector (X). The Euclidean norm is used to normalize the size of the vector to make the inner product results more comparable.

[0130] Then, the Euclidean norm (|Y|) of the radar feature vector is calculated. The calculation formula of the Euclidean norm is also (|Y|=sqrt{sum_{i=1}^{n}y_i^2}), which represents the length of the feature vector (Y). The role of the Euclidean norm is the same as the above step, which is used to normalize the size of the radar feature vector.

[0131] Finally, based on the inner product and the norm, the correlation coefficient (R_{xy}) is calculated. The calculation formula of the correlation coefficient is (R_{xy}=frac{XcdotY}{|X||Y|}). In this formula, (XcdotY) represents the inner product of the image feature vector and the radar feature vector, and (|X|) and (|Y|) represent the Euclidean norm of the image feature vector and the radar feature vector, respectively. This formula normalizes the inner product result by dividing the inner product by the product of the lengths of the two vectors, so that the correlation coefficient ranges from ([-1,1]). When (R_{xy}) is close to 1, it means that the two feature vectors are highly correlated; when (R_{xy}) is close to -1, it means that the two feature vectors are negatively correlated; when (R_{xy}) is close to 0, it means that the two feature vectors are uncorrelated.

[0132] For example, in one embodiment, assume that there is a target detection task in which both the image feature vector (X) and the radar feature vector (Y) are 3-dimensional. Specifically, (X = [1, 2, 3]), (Y = [2, 4, 6]). First, the inner product (XcdotY = 1times2+2times4+3times6 = 28) is calculated. Next, the Euclidean norm of the image feature vector (|X| = sqrt{1^2+2^2+3^2} = sqrt{14}) and the Euclidean norm of the radar feature vector are calculated.

[0133] (|Y|=sqrt{2^2+4^2+6^2}=sqrt{56}). Finally, the correlation coefficient is calculated (R_{xy}=frac{28}{sqrt{14}timessqrt{56}}=frac{28}{28}=1). This shows that the image feature vector and the radar feature vector are highly correlated.

[0134] Next, the step of the present invention based on the inner product (XcdotY) of the image feature vector and the radar feature vector is described. This step involves element-by-element multiplication of the image feature vector and the radar feature vector, then summing up all the product results, and finally judging the similarity of the two feature vectors based on the results.

[0135] First, the corresponding components of the image feature vector (X) and the radar feature vector (Y) are multiplied. Specifically, assuming that (X = [x_1, x_2, ..., x_n]) and (Y = [y_1, y_2, ..., y_n]) are two feature vectors of length (n), the purpose of this step is to calculate (x_1cdoty_1, x_2cdoty_2, ..., x_ncdoty_n). These products reflect the degree of correlation between the image features and the radar features in each feature dimension.

[0136] Next, we sum the product of all components. This means calculating (XcdotY = x_1cdoty_1 + x_2cdoty_2 + ... + x_ncdoty_n). This step generates a single value by accumulating all the products, which represents the overall similarity or correlation between the two feature vectors.

[0137] When the inner product (XcdotY) is greater than the preset threshold (T), the feature vector is considered to have a high similarity. The formula here is (XcdotY>T), where (XcdotY) represents the inner product and (T) is the preset threshold. The selection of parameter (T) needs to be adjusted according to the actual application scenario, and the range is usually between 0 and a positive value. The optimal value depends on the specific data set and target detection requirements. If the inner product value is greater than (T), it means that the image feature vector and the radar feature vector have a strong correlation in multiple dimensions, which is helpful for subsequent target detection.

[0138] The inner product result is used for subsequent correlation calculation. In one embodiment, assume that there is a target detection system, the feature vector in a certain image is (X = [0.9, 0.7, 0.4, 0.5]), the feature vector corresponding to the radar data is (Y = [0.8, 0.6, 0.3, 0.5]), and the preset threshold (T) is 1.2. According to the above steps, first calculate the corresponding component product of (X) and (Y):

[0139] (0.9cdot0.8=0.72, 0.7cdot0.6=0.42, 0.4cdot0.3=0.12, 0.5cdot0.5=0.25). Then, these product results are summed to obtain the inner product (XcdotY=0.72+0.42+0.12+0.25=1.51). Since (1.51>1.2), it can be considered that the image feature vector and the radar feature vector have a high similarity in this image. This result can be further used for target detection to improve the accuracy and robustness of the system.

[0140] Next, the steps of the present invention based on summing the product results of all components are described.

[0141] The first step is to initialize an accumulation variable S to 0. This means that before starting the process, the accumulation variable S is set to zero to store the intermediate results of subsequent calculations.

[0142] The second step is to traverse the corresponding components of the image feature vector and the radar feature vector and calculate the product of each component. In this process, the system will compare each corresponding component in the image feature vector and the radar feature vector one by one and calculate the product between them. This is to quantify the similarity between the two different modal data. By fusing the image information with the radar information, the target object can be identified more accurately.

[0143] The next step is to accumulate the product of each group of components into the cumulative variable S. After calculating the product of each pair of components in the previous step, the system accumulates it into the cumulative variable S that has been initialized to zero. The purpose of this step is to aggregate all similarity scores to form a comprehensive measurement indicator.

[0144] Finally, when the cumulative variable S is greater than the threshold T, the feature vector is considered to have a high similarity. The formula is (S>T), where S represents the cumulative variable and T is the preset threshold. The threshold T is usually set according to the specific application scenario, the range can be adjusted according to the actual situation, and the optimal value needs to be determined through experiments. The significance of this formula is to determine whether the overall similarity between two feature vectors reaches or exceeds a certain standard, thereby deciding whether to identify these features as an important basis for target detection. In the target detection task, this step helps to eliminate noise interference and ensure the reliability of the detection results.

[0145] Specifically, in one embodiment, assuming that the image feature vector is ([0.3, 0.6, 0.2]) and the radar feature vector is ([0.8, 0.5, 0.7]), the calculation process is as follows: first, initialize the accumulation variable S to 0; then calculate the product of each pair of components: (0.3 times 0.8 = 0.24), (0.6 times 0.5 = 0.3), (0.2 times 0.7 = 0.14); then add these products to S: (0.24 + 0.3 + 0.14 = 0.68). If the preset threshold T is 0.5, then at this time (S = 0.68> 0.5), it is considered that the similarity between the image and the radar feature vector is high and may correspond to the same target. In this way, the accuracy of target detection can be improved by combining visual and radar data.

[0146] A cross-modal adaptive feature extraction method for target detection of the present invention includes the following main steps:

[0147] 1. Constructing the initial multimodal feature representation: First, construct the initial multimodal feature representation based on multiple modal data (such as images, videos, text, etc.). This step ensures that data of different modalities can be effectively represented as feature vectors for subsequent processing.

[0148] 2. Preliminary fusion processing: Perform preliminary fusion processing on the constructed multimodal feature representation so that the features of each modality can be compared and processed on the same dimension. This step helps to solve the alignment difficulty caused by the difference in feature representation between different modal features to a certain extent.

[0149] 3. The cross-modal adaptive layer adjusts the weights of each modal feature: By introducing the cross-modal adaptive layer, the layer can automatically learn and adjust the weights of each modal feature. This process solves the problem of data bias caused by the imbalanced distribution of features between different modalities, avoids the problem of the model being biased towards a certain modal feature, and thus improves the robustness and accuracy of detection.

[0150] 4. Input weighted features into the detection model: Input the weighted multimodal features into the target detection model for target detection. This step ensures the effective integration and utilization of each modality feature, while improving the accuracy and speed of detection.

[0151] 5. Detection result feedback optimization: According to the feedback of target detection results, the parameters of the cross-modal adaptive layer are dynamically optimized. Through continuous feedback and optimization, the model can better adapt to the changes of different modal data and further improve the detection performance.

[0152] To solve specific technical problems, the approach adopts the following strategies:

[0153] 1. Cross-modal alignment problem: Through preliminary fusion processing and weight adjustment of the cross-modal adaptive layer, this method can establish effective alignment between different modalities and reduce the difficulty of alignment caused by differences in feature representation.

[0154] 2. Unbalanced feature distribution problem: The cross-modal adaptive layer dynamically adjusts the weights of each modal feature, which can effectively deal with the imbalance of feature distribution between different modalities and avoid the model's excessive dependence on a certain modality.

[0155] 3. The problem of too high feature dimension: Through preliminary fusion processing, multimodal features are mapped to the same dimension, which reduces the feature dimension and improves computational efficiency.

[0156] 4. Data quality issue: The cross-modal adaptive layer can automatically reduce the weight of low-quality modal features, retain the information of high-quality features, and reduce the problem of feature information loss caused by poor data quality.

[0157] 5. Noise interference problem: Dynamically optimize the parameters of the cross-modal adaptive layer, and continuously adjust the weights according to the detection result feedback to reduce the impact of noise modal characteristics and prevent overfitting. The combined effect of these technical means enables the present invention to effectively cope with various challenges in multimodal target detection.

[0158] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium.

[0159] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0160] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0161] Any process or method description in the flow chart or otherwise described herein can be understood to represent a module, fragment or portion of a code including one or more executable instructions for implementing the steps of a specific logical function or process. And the scope of the preferred embodiment of the present application includes other implementations, in which the functions may not be performed in the order shown or discussed, including in a substantially simultaneous manner or in a reverse order according to the functions involved.

[0162] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, which can be embodied in any computer-readable medium for use by an instruction execution system, apparatus or device (such as a computer-based system, a system including a processor or other system that can fetch instructions from an instruction execution system, apparatus or device and execute instructions), or used in combination with these instruction execution systems, apparatuses or devices.

[0163] It should be understood that the various parts of the present application can be implemented with hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented with software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above embodiment method can be completed by instructing the relevant hardware through a program, which can be stored in a computer-readable storage medium, and when the program is executed, it includes one of the steps of the method embodiment or a combination thereof.

[0164] In addition, each functional unit in each embodiment of the present application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the above-mentioned integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium can be a read-only memory, a disk or an optical disk, etc.

[0165] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A cross-modal adaptive feature extraction method for target detection, characterized in that: The following steps are involved: Based on multi-modal data, construct initial multi-modal feature representation; Performing preliminary fusion processing on the multimodal feature representation; Through the cross-modal adaptation layer, the weights of each modality feature are adjusted; Input the weighted features into the detection model for target detection; According to the detection result feedback, the parameters of the cross-modal adaptation layer are optimized.

2. The cross-modal adaptive feature extraction method for target detection according to claim 1, characterized in that: Based on multi-modal data, the steps of constructing the initial multi-modal feature representation include: Collect data from at least two different modalities, such as imagery and radar data; Preprocess the data of each modality, including data normalization and noise filtering; Based on the preprocessed data, extract the respective feature representations; The feature representations of all modalities are preliminarily integrated to form an initial multimodal feature representation.

3. The cross-modal adaptive feature extraction method for target detection according to claim 2, characterized in that: Based on the preprocessed data, the steps of extracting the respective feature representations include: Perform convolutional neural network feature extraction on image data to obtain image feature vectors; Perform Fourier transform on radar data to obtain radar spectrum characteristics; Based on the eigenvectors of the two modal data, the correlation of the eigenvectors is calculated; When the correlation coefficient is greater than the threshold C, the two modal data are selected for subsequent processing, and the formula is R_{xy}>C, where R_{xy} represents the correlation coefficient between the image feature vector and the radar feature vector, and C is the preset threshold.

4. The cross-modal adaptive feature extraction method for target detection according to claim 3, characterized in that: Based on the eigenvectors of the two modal data, the steps of calculating the correlation of the eigenvectors include: Calculate the dot product of the image feature vector and the radar feature vector; Calculate the norms of the two eigenvectors; Based on the dot product and norm, the correlation coefficient R_{xy}=frac{XcdotY}{|X||Y|} is calculated, where X and Y are the image feature vector and radar feature vector respectively; When the correlation coefficient R_{xy} is greater than the threshold C, the two modal data are selected for subsequent processing, and the formula is R_{xy}>C, where R_{xy} represents the correlation coefficient between the image feature vector and the radar feature vector, and C is the preset threshold.

5. The cross-modal adaptive feature extraction method for target detection according to claim 4, characterized in that: Based on the dot product and the norm, the steps of calculating the correlation coefficient R_{xy}=frac{XcdotY}{|X||Y|} include: Calculate the inner product (XcdotY) of the image feature vector and the radar feature vector; Calculate the Euclidean norm of the image feature vector (|X|); Calculate the Euclidean norm (|Y|) of the radar feature vector; Based on the inner product and the norm, the correlation coefficient R_{xy}=frac{XcdotY}{|X||Y|} is calculated.

6. The method for optimizing the large model network retrieval capability as claimed in claim 5, characterized in that: The steps based on the inner product (XcdotY) of the image feature vector and the radar feature vector include: Multiply the corresponding components of the image feature vector and the radar feature vector; Sum the products of all components; When the inner product XcdotY is greater than the threshold T, the feature vector is considered to have a high similarity. The formula is XcdotY>T, where XcdotY represents the inner product and T is the preset threshold. The inner product result is used for subsequent correlation calculation.

7. The cross-modal adaptive feature extraction method for target detection according to claim 3, characterized in that: The steps based on summing the product of all components include: Initialize a cumulative variable S to 0; Traverse the corresponding components of the image feature vector and the radar feature vector, and calculate the product of each group of components; Add the product of each group's weight to the cumulative variable S; When the cumulative variable S is greater than the threshold T, the feature vector is considered to have a higher similarity, and the formula is S>T, where S represents the cumulative variable and T is the preset threshold.

Images