A target matching method, electronic device, and storage medium

By performing feature alignment and fusion in the cross-modal target matching task and combining multiple alignment methods, the problem of misalignment of features in different modal data is solved, and the accuracy of target matching and the performance of cross-modal detection is improved.

CN118690334BActive Publication Date: 2024-11-29ZHEJIANG DAHUA TECH CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202411177401.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-26
Publication Date
2024-11-29
Estimated Expiration
2044-08-26

AI Technical Summary

Technical Problem

In the cross-modal target matching task, due to the significant distribution gap between different modal data, the extracted features are not aligned, which in turn affects the matching accuracy of multimodal target detection.

Method used

By obtaining target data of different modes, extracting the features of its sub-target data, and performing feature alignment and fusion, calculating similarity using mapping features and alignment features, combining multiple alignment methods (original alignment, fusion alignment, interpolation alignment) to Reduce feature differences and adjust network parameters of the target matching model to improve matching accuracy.

Benefits of technology

It effectively reduces the differences between multimodal data features, improves the accuracy of target matching, and enhances the performance of cross-modal object detection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118690334B_ABST
    Figure CN118690334B_ABST
Patent Text Reader

Abstract

The present application discloses a target matching method, an electronic device, and a storage medium, the method comprising: obtaining first target data and second target data of different modalities; extracting the first target feature corresponding to the first sub-target data in the first target data, and extracting the second target feature corresponding to the second sub-target data in the second target data; obtaining a first similarity based on the feature alignment result between the first target feature and the second target feature, and obtaining a second similarity based on the feature fusion result between the first target feature and the second target feature; obtaining a matching result that characterizes whether the first sub-target data and the second sub-target data have the same meaning based on the fusion similarity of the first similarity and the second similarity. The above scheme can improve the accuracy of target matching.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of target detection, and in particular to a target matching method, electronic equipment, and storage medium. Background Art

[0002] The cross-modal target matching task mainly combines multiple different modal information to perform target matching tasks. When performing cross-modal target matching, features of data of different modalities are extracted. However, due to the significant distribution gap between different modal data in the prior art, the extracted features of different modalities are not aligned, resulting in incorrect feature matching during multi-modal target detection, which is not conducive to cross-modal target matching. Summary of the invention

[0003] The present application at least provides a target matching method, an electronic device, and a storage medium, which can reduce the differences between multimodal data features to improve the accuracy of target matching.

[0004] In a first aspect, the present application provides a target matching method, which includes: acquiring first target data and second target data of different modalities, wherein the modalities of the first target data and the second target data are any two of image, text, audio, and video; extracting a first target feature corresponding to the first sub-target data in the first target data, and extracting a second target feature corresponding to the second sub-target data in the second target data; obtaining a first similarity based on a feature alignment result between the first target feature and the second target feature, and obtaining a second similarity based on a feature fusion result between the first target feature and the second target feature; obtaining a matching result characterizing whether the first sub-target data and the second sub-target data have the same meaning based on the fused similarity of the first similarity and the second similarity.

[0005] Wherein, based on the feature alignment result between the first target feature and the second target feature, the first similarity is obtained, including: based on the first target feature, mapping to obtain a first mapping feature, and based on the second target feature, mapping to obtain a second mapping feature of the same dimension as the first mapping feature; obtaining a first alignment feature and a second alignment feature after the first mapping feature and the second mapping feature are respectively processed by the same alignment method; based on the first alignment feature and the second alignment feature, the first similarity is obtained.

[0006] Among them, the first similarity includes at least one of original alignment similarity, fused alignment similarity, and interpolation alignment similarity; among them, the original alignment similarity is measured when the alignment mode is original alignment, the fused alignment similarity is measured when the alignment mode is fused alignment, and the interpolation alignment similarity is measured when the alignment mode is interpolation alignment.

[0007] Among them, the first similarity at least includes the original alignment similarity, the alignment method at least includes the original alignment, and the first alignment feature and the second alignment feature are obtained after the first mapping feature and the second mapping feature are processed by the same alignment method respectively, including: directly selecting the first mapping feature as the first alignment feature after the original alignment processing, and directly selecting the second mapping feature as the second alignment feature after the original alignment processing.

[0008] Among them, the first similarity at least includes fusion alignment similarity, the alignment method at least includes fusion alignment, and the first alignment feature and the second alignment feature are obtained after the first mapping feature and the second mapping feature are processed by the same alignment method respectively, including: based on the first mapping feature and the second mapping feature, they are fused in a first order to obtain the first alignment feature, and based on the first mapping feature and the second mapping feature, they are fused in the reverse order of the first order to obtain the second alignment feature.

[0009] Among them, based on the first mapping feature and the second mapping feature, the first alignment feature is obtained by fusing them in a first order, including: splicing the first mapping feature before the second mapping feature to obtain the first alignment feature; based on the first mapping feature and the second mapping feature, the second alignment feature is obtained by fusing them in the reverse order of the first order, including: splicing the first mapping feature after the second mapping feature to obtain the second alignment feature.

[0010] Among them, the first similarity at least includes interpolation alignment similarity, the alignment method at least includes interpolation alignment, and obtaining the first alignment feature and the second alignment feature after the first mapping feature and the second mapping feature are respectively processed by the same alignment method, including: interpolating in a second order based on the first mapping feature and the second mapping feature to obtain the first alignment feature, and interpolating in an opposite order of the second order based on the first mapping feature and the second mapping feature to obtain the second alignment feature.

[0011] Among them, the matching result is predicted by the target matching model, and the training method of the target matching model includes: extracting the first sample data feature of the first sample data, and extracting the second sample data feature of the second sample data, wherein the modalities of the first sample data and the second sample data are any two of image, text, audio, and video; based on the first sample data feature, mapping to obtain the first sample mapping feature, and based on the second sample data feature, mapping to obtain the second sample mapping feature of the same dimension as the first sample mapping feature; based on the size relationship of the target feature difference between the first sample mapping feature and the second sample mapping feature relative to the target difference range, obtaining the training loss; at least based on the training loss, adjusting the network parameters of the target matching model.

[0012] Among them, before obtaining the training loss based on the size relationship of the target feature difference between the first sample mapping feature and the second sample mapping feature relative to the target difference range, the method also includes: obtaining the first sample alignment feature and the second sample alignment feature after the first sample mapping feature and the second sample mapping feature are respectively processed by the same alignment method; using multiple measurement methods to measure the difference between the first sample alignment feature and the second sample alignment feature, and obtaining sub-feature differences of various measurement methods, wherein the multiple measurement methods include at least: Euclidean distance, cosine distance; based on the sub-feature differences of various measurement methods, obtaining the target feature difference.

[0013] The alignment method includes original alignment, the training loss includes original alignment sub-loss, and the original alignment sub-loss is obtained by the target feature difference between the first sample alignment feature and the second sample alignment feature after the original alignment process.

[0014] The alignment method includes fusion alignment, the training loss includes fusion alignment sub-loss, and the fusion alignment sub-loss is obtained by the target feature difference between the first sample alignment feature and the second sample alignment feature after the fusion alignment process.

[0015] The alignment method includes interpolation alignment, the training loss includes an interpolation alignment sub-loss, and the interpolation alignment sub-loss is obtained by a target feature difference between a first sample alignment feature and a second sample alignment feature after the interpolation alignment process.

[0016] Among them, based on the size relationship of the target feature difference between the first sample mapping feature and the second sample mapping feature relative to the target difference range, the training loss is obtained, including: in response to the target feature difference being within the target difference range, the sum of the sub-feature differences of various metrics is obtained as the fused feature difference, and the target feature difference is used as the base and the preset hyperparameter is used as the exponent to obtain the weight parameter; based on the fused feature difference and the weight parameter, the training loss is obtained.

[0017] A second aspect of the present application provides an electronic device, including a memory and a processor coupled to each other, wherein the processor is used to execute program instructions stored in the memory to implement the target matching method in the first aspect.

[0018] A third aspect of the present application provides a computer-readable storage medium having program instructions stored thereon, and when the program instructions are executed by a processor, the target matching method in the first aspect is implemented.

[0019] The above scheme obtains the first target data and the second target data of different modalities, and performs feature extraction on the first sub-target data in the first target data to obtain the first target feature corresponding to the first sub-target data, and performs feature extraction on the second sub-target data in the second target data to obtain the second target feature corresponding to the second sub-target data, and performs feature alignment on the first target feature and the second target feature and calculates the similarity to obtain the first similarity, and at the same time performs feature fusion on the first target feature and the second target feature and calculates the similarity to obtain the second similarity, and uses the fused similarity of the first similarity and the second similarity to obtain a matching result that characterizes whether the first sub-target data and the second sub-target data have the same meaning, and based on the similarity after the fusion of the first similarity and the second similarity, it can be accurately determined whether the meanings expressed by the sub-data of data in different modalities are the same, thereby improving the accuracy of target matching.

[0020] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0022] Figure 1 It is a flowchart of an embodiment of the target matching method of the present application;

[0023] Figure 2 It is a structural framework diagram of an embodiment of the target matching model of the present application;

[0024] Figure 3 It is a flowchart of an embodiment of a training method for a target matching model of the present application;

[0025] Figure 4 It is a flowchart of an embodiment of a target detection network of the present application;

[0026] Figure 5 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application;

[0027] Figure 6 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0028] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0029] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0030] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the objects associated before and after are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.

[0031] See also Figure 1 , Figure 1 This is a flow chart of an embodiment of the target matching method of the present application. Specifically, it may include the following steps:

[0032] Step S110: Acquire first target data and second target data of different modalities.

[0033] The modalities of the first target data and the second target data are any two of image, text, audio and video.

[0034] This application is mainly used in the field of cross-modal target detection. In multimodal target detection, the distribution gap between different modal data is significantly reduced, so that the extracted features of different modalities are aligned, so that the results of multimodal data matching are more accurate.

[0035] In some embodiments, the target data may be image data obtained by taking a camera, audio data obtained by recording a voice recorder, or text data obtained by keyboard input. In addition to the above-mentioned real-time acquisition method, the target data may also be obtained from a database, which is not specifically limited here. Among them, the target data is the first target data and the second target data.

[0036] Step S120: extracting the first target feature corresponding to the first sub-target data in the first target data, and extracting the second target feature corresponding to the second sub-target data in the second target data.

[0037] In some embodiments, in order to make the target matching result more accurate, the first target data and the second target data may be split before feature extraction to obtain a plurality of first sub-target data and a plurality of second sub-target data. For example, if the target data obtained is a picture, a sliding window algorithm may be used to split the picture; if the target data obtained is text, the text may be segmented or segmented. It is understandable that targeted segmentation methods may be used for data of different modalities, and no specific limitation is made here.

[0038] After obtaining a plurality of first sub-target data and a plurality of second sub-target data, feature extraction is performed on the first sub-target data and the second sub-target data respectively to obtain first target features corresponding to the first sub-target data and second target features corresponding to the second sub-target data.

[0039] Specifically, please refer to Figure 2 The target matching model 200 in this application is mainly used in the field of cross-modal target detection, and is used to detect and match data of different modalities, and to determine whether the meanings represented by the data of different modalities are the same through the matching results. Therefore, the feature extraction network 210 in the target matching model 200 can be used to split the first target data and the second target data to obtain a number of first sub-target data and a number of second sub-target data, and then feature extraction is performed on the first sub-target data and the second sub-target data respectively to obtain the first target feature corresponding to the first sub-target data and the second target feature corresponding to the second sub-target data.

[0040] Step S130: obtaining a first similarity based on a feature alignment result between the first target feature and the second target feature, and obtaining a second similarity based on a feature fusion result between the first target feature and the second target feature.

[0041] In some embodiments, after obtaining the first target feature and the second target feature, feature alignment may be performed on the first target feature and the second target feature to obtain a first similarity. Specifically, refer to step S131 to step S133.

[0042] Step S131: Based on the first target feature, a first mapping feature is mapped, and based on the second target feature, a second mapping feature having the same dimension as the first mapping feature is mapped.

[0043] Please continue reading Figure 2 , the mapping module 220 in the target matching model 200 can be used to map the first target feature into the mapping space to obtain the first mapping feature; and then the mapping module 220 can be used to map the second target feature into the same mapping space as the first mapping feature to obtain the second mapping feature, so that the first mapping feature and the second mapping feature are located in the same mapping space, and therefore the dimensions of the first mapping feature and the second mapping feature will also be the same.

[0044] Step S132: obtaining a first alignment feature and a second alignment feature after the first mapping feature and the second mapping feature are processed by the same alignment method respectively.

[0045] In some embodiments, the first mapping feature and the second mapping feature may be aligned using the same alignment method, for example, original alignment, fusion alignment, interpolation alignment, etc., which are not specifically limited here.

[0046] In a specific embodiment, the alignment module 230 in the target matching model 200 includes three alignment modes: original alignment, fusion alignment, and interpolation alignment. Therefore, the alignment module 230 can be used to perform feature alignment on the first mapping feature and the second mapping feature.

[0047] When the alignment mode adopted by the alignment module 230 is original alignment, the first mapping feature may be directly selected as the first alignment feature after the original alignment process, and the second mapping feature may be directly selected as the second alignment feature after the original alignment process.

[0048] When the alignment mode adopted by the alignment module 230 is fusion alignment, the first mapping feature and the second mapping feature can be fused in a first order to obtain a first alignment feature, and the first mapping feature and the second mapping feature can be fused in the reverse order of the first order to obtain a second alignment feature.

[0049] Specifically, the first order may be to splice the first mapping feature before the second mapping feature to obtain the first alignment feature, and then to splice the first mapping feature after the second mapping feature to obtain the second alignment feature.

[0050] When the alignment mode adopted by the alignment module 230 is interpolation alignment, interpolation can be performed based on the first mapping feature and the second mapping feature in the second order to obtain the first alignment feature, and interpolation can be performed based on the first mapping feature and the second mapping feature in the reverse order of the second order to obtain the second alignment feature.

[0051] Step S133: obtaining a first similarity based on the first alignment feature and the second alignment feature.

[0052] In some embodiments, the first similarity includes at least one of original alignment similarity, fused alignment similarity, and interpolated alignment similarity. The original alignment similarity is measured when the alignment mode is original alignment, the fused alignment similarity is measured when the alignment mode is fused alignment, and the interpolated alignment similarity is measured when the alignment mode is interpolated alignment.

[0053] Therefore, when the first similarity includes at least the original alignment similarity, the alignment mode includes at least the original alignment; when the first similarity includes at least the fused alignment similarity, the alignment mode includes at least the fused alignment; when the first similarity includes at least the interpolated alignment similarity, the alignment mode includes at least the interpolated alignment.

[0054] Specifically, the first similarity may include the original alignment similarity, the fused alignment similarity, and the interpolated alignment similarity at the same time, and the original alignment similarity, the fused alignment similarity, and the interpolated alignment similarity are added together to obtain the first similarity.

[0055] In some embodiments, the first target feature and the second target feature may be weightedly fused to obtain a feature fusion result. The feature fusion result is then calculated to obtain a second similarity. Specifically, the feature fusion result may be analyzed using a logic function. If the feature fusion result is greater than or equal to a preset logic value, the logic function outputs 1, i.e., 1 is the second similarity; if the feature fusion result is less than the preset logic value, the logic function outputs 0, i.e., 0 is the second similarity. It is understandable that the method for analyzing the feature fusion result is not specifically limited herein.

[0056] It is understandable that the calculation method of the first similarity and the second similarity may also adopt methods such as cosine similarity and Pearson similarity, which are not specifically limited here.

[0057] Step S140: Based on the fused similarity of the first similarity and the second similarity, a matching result is obtained, which indicates whether the first sub-target data and the second sub-target data have the same meaning.

[0058] In some embodiments, the first similarity and the second similarity may be multiplied or added to obtain a matching result, and the matching result is compared with a preset target threshold. If the matching result is greater than or equal to the preset target threshold, the first sub-target data and the second sub-target data represent the same meaning; if the matching result is less than the preset target threshold, the first sub-target data and the second sub-target data represent different meanings. It is understandable that the fusion method of the first similarity and the second similarity may also be weighted fusion, addition, etc., which is not specifically limited here.

[0059] In order to make the final target matching degree more accurate, the target matching model 200 can be trained before application. After the training result of the target matching model 200 reaches the expected level, the target matching model 200 is officially put into use, and the adjustment module 240 in the target matching model 200 is only used during the training stage. For the training method of the target matching model 200, please refer to the following steps.

[0060] See also Figure 3 , Figure 3 This is a flow chart of an embodiment of a training method for a target matching model of the present application. Specifically, the following steps may be included:

[0061] Step S310: extracting first sample data features of first sample data, and extracting second sample data features of second sample data.

[0062] The modalities of the first sample data and the second sample data are any two of image, text, audio, and video. It is understandable that the first sample data and the second sample data may be other modal data in addition to the above four modal data, which is not specifically limited here.

[0063] Furthermore, the first sample data in the present application can be taken from any first modal data, and the second sample data can be taken from any second modal data. The target matching model 200 extracts features from different modal data to determine whether the different modal data represent the same meaning. In the training process, the first sample data and the second sample data can be annotated with sample tags for identifying whether they have the same meaning.

[0064] For example, the first sample data is a picture of a puppy, and the second sample data is a text describing a flower. The sample annotations in the first sample data and the second sample data indicate that their meanings are different. During the training process, the target matching model 200 is used to extract features from the first sample data and the second sample data and output a target detection result. If the output target detection result shows that the meanings represented by the first sample data and the second sample data are inconsistent, it indicates that the target detection effect of the target matching model 200 meets expectations; otherwise, the network parameters of the target matching model 200 are adjusted.

[0065] For another example, the first sample data is a picture of a puppy, and the second sample data is a text describing the dog. The sample annotations in the first sample data and the second sample data indicate that the two have the same meaning. During the training process, the target matching model 200 is used to extract features from the first sample data and the second sample data and output a target detection result. If the output target detection result shows that the meanings represented by the first sample data and the second sample data are consistent, it indicates that the target detection effect of the target matching model 200 meets expectations. Otherwise, the parameters of the target matching model 200 are adjusted.

[0066] In some embodiments, the target matching model 200 may include a feature extraction network 210. After the first sample data and the second sample data are input into the target matching model 200, they are vectorized by the target matching model 200, and then the vectorized first sample data and the second sample data are input into the feature extraction network 210 for feature extraction to obtain first sample data features and second sample data features.

[0067] Step S320: Based on the first sample data feature, a first sample mapping feature is mapped, and based on the second sample data feature, a second sample mapping feature having the same dimension as the first sample mapping feature is mapped.

[0068] It can be seen from the above content that the first sample data feature and the second sample data feature are derived from the first sample data and the second sample data of different modalities, and the dimensions of the features obtained according to the different modal data may be different, which will seriously affect the target detection effect in the subsequent target detection process. Therefore, in order to reduce the impact of the different dimensions of the features of different modal data, the first sample data feature and the second sample data feature need to be converted so that the dimensions of the first sample data feature and the second sample data feature are the same. In this embodiment, the first sample data feature can be mapped to obtain a first sample mapping feature, and then the second sample data feature can be mapped to obtain a second sample mapping feature with the same dimension as the first sample mapping feature.

[0069] Specifically, the target matching model 200 may include a mapping module 220, which is used to map the first sample data feature into a mapping space to obtain a first sample mapping feature; and then the mapping module 220 is used to map the second sample data feature into the same mapping space as the first sample mapping feature to obtain a second sample mapping feature, so that the first sample mapping feature and the second sample mapping feature are located in the same mapping space, and therefore the dimensions of the first sample mapping feature and the second sample mapping feature will also be the same.

[0070] Step S330: obtaining a training loss based on a magnitude relationship of a target feature difference between the first sample mapping feature and the second sample mapping feature relative to a target difference range.

[0071] In some embodiments, there are essential differences in feature distribution between the first sample mapping feature and the second sample mapping feature of different modalities, and each modality has its own uniqueness. Therefore, in order to align the first sample mapping feature and the second sample mapping feature, in the present application, the target feature difference between the first sample mapping feature and the second sample mapping feature is restricted. If the target feature difference between the first sample mapping feature and the second sample mapping feature is within the target difference range, it is indicated that the first sample mapping feature and the second sample mapping feature are aligned.

[0072] Regarding the method of obtaining the target feature difference between the first sample mapping feature and the second sample mapping feature, please refer to step S331 to step S333.

[0073] Step S331: obtaining the first sample alignment feature and the second sample alignment feature after the first sample mapping feature and the second sample mapping feature are processed by the same alignment method respectively.

[0074] The alignment method may be original alignment, fusion alignment, etc., which is not specifically limited here. Step S332: using multiple measurement methods to measure the difference between the first sample alignment feature and the second sample alignment feature, respectively, to obtain sub-feature differences of various measurement methods.

[0075] In some embodiments, multiple metrics may be calculated on the aligned first sample alignment feature and the second sample alignment feature to obtain sub-feature differences. For example, the difference between the first sample alignment feature and the second sample alignment feature may be calculated using metrics such as Euclidean distance and cosine distance.

[0076] Step S333: Obtain target feature differences based on sub-feature differences in various measurement methods.

[0077] In some embodiments, the sub-feature differences of various metrics may be weighted and summed to obtain the target feature difference.

[0078] In addition, the target feature difference can also be obtained according to the ratio of the sub-feature differences in various measurement methods. Therefore, the method for obtaining the target feature difference is not specifically limited here.

[0079] In some embodiments, when the alignment method includes raw alignment, the training loss includes a raw alignment sub-loss, and the raw alignment sub-loss is obtained by the target feature difference between the first sample alignment feature and the second sample alignment feature after the raw alignment process.

[0080] In some other embodiments, when the alignment method includes fusion alignment, the training loss includes a fusion alignment sub-loss, and the fusion alignment sub-loss is obtained by the target feature difference between the first sample alignment feature and the second sample alignment feature after the fusion alignment process.

[0081] In some other embodiments, when the alignment method includes interpolation alignment, the training loss includes an interpolation alignment sub-loss, and the interpolation alignment sub-loss is obtained by the target feature difference between the first sample alignment feature and the second sample alignment feature after the interpolation alignment process.

[0082] In some embodiments, after obtaining the target feature difference between the first sample alignment feature and the second sample alignment feature, the training loss can be determined according to whether the target feature difference is within the target difference range. Specifically, in response to the target feature difference being within the target difference range, the sum of the sub-feature differences of various metrics is obtained as the fused feature difference, and the target feature difference is used as the base and the preset hyperparameter is used as the exponent to obtain the weight parameter, and then the training loss is obtained according to the fused feature difference and the weight parameter.

[0083] In addition, in response to the target feature difference being outside the target difference range, a preset value is obtained as the training loss; wherein the preset value is not greater than the minimum training loss when the target feature difference is within the target difference range.

[0084] In a specific embodiment, the training loss is a ring feature alignment loss function, and the specific formula of the ring feature alignment loss function is:

[0085] (1)

[0087] The target difference range is , is the first sample alignment feature after alignment processing, is the alignment feature of the second sample after alignment processing, is the sub-feature difference of Euclidean distance, is the sub-feature difference of cosine distance, To preset hyperparameters, is the preset value, is the weight parameter, s is the similarity of two features, so s is defined as: .

[0088] According to the above ring feature alignment loss function It can be seen that the target feature difference between the first sample alignment feature and the second sample alignment feature after the two modal data are aligned is less than the threshold value. , the two modal data have been aligned, so for the first sample data and the second sample data, the ring feature alignment loss function The corresponding restrictions will be adaptively ignored. When the target feature difference between the first sample alignment feature and the second sample alignment feature after the two modal data are aligned is very large and greater than another threshold value , it can be considered that at the current training iteration, the features corresponding to the two modal data are completely unrelated. Therefore, it is obviously unreasonable to force the alignment of the first sample alignment features and the second sample alignment features, which are completely unrelated, to align at this moment. Therefore, the circular feature alignment loss function The constraints on the first sample data and the second sample data will also be ignored in the current training. As the training progresses, the coupling between the first sample data and the second sample data will become stronger and stronger. After they are no longer unlearnable samples, the ring feature alignment loss function can be used. At the same time, this application introduces This weight parameter gives a larger weight to samples with greater feature differences in modal data within a specified range, thereby focusing on strengthening the feature alignment of samples with large differences in target features of modal data.

[0089] Please continue reading Figure 2 In order to achieve the alignment of the first sample mapping feature and the second sample mapping feature, an alignment module 230 can be set in the target matching model 200. The alignment module 230 can perform three alignment methods: original alignment, fusion alignment and interpolation alignment. Among them, the original alignment method aligns the first sample mapping feature and the second sample mapping feature from the original feature aspect, the fusion alignment method aligns the first sample mapping feature and the second sample mapping feature from the overall expression of the feature, and the interpolation alignment method aligns the first sample mapping feature and the second sample mapping feature from the local distribution of the feature channel, thereby narrowing the feature distribution difference between the two modalities to further improve the detection accuracy of the target matching model 200.

[0090] In some embodiments, when the alignment module 230 runs the original alignment mode, the training loss obtained by the target matching model 200 includes the original alignment sub-loss. Specifically, the original alignment mode directly selects the first sample mapping feature as the first sample alignment feature after the original alignment process, and directly selects the second sample mapping feature as the second sample alignment feature after the original alignment process. The original alignment sub-loss is obtained by the target feature difference between the first sample alignment feature and the second sample alignment feature after the original alignment process.

[0091] For example, the above ring feature alignment loss function can be used Formula (1) calculates the original alignment loss, is the first sample mapping feature, Map features for the second sample:

[0092] (2)

[0093] in, is the first sample alignment feature after the original alignment process, is the second sample alignment feature after the original alignment process, i=1,…,n.

[0094] The original alignment method can be used to converge the first sample mapping features and the second sample mapping features during the alignment process during training.

[0095] In other embodiments, when the alignment module 230 runs a fusion alignment mode, the training loss obtained by the target matching model 200 includes a fusion alignment sub-loss. Specifically, the fusion alignment mode is based on the first sample mapping feature and the second sample mapping feature being fused in a first order to obtain a first sample alignment feature, and based on the first sample mapping feature and the second sample mapping feature being fused in the reverse order of the first order to obtain a second sample alignment feature. The fusion alignment sub-loss is obtained by the target feature difference between the first sample alignment feature and the second sample alignment feature after the fusion alignment process.

[0096] Furthermore, the first order may be that the first sample mapping feature is before the second sample mapping feature. For example, the first sample mapping feature is spliced ​​before the second sample mapping feature to obtain the first sample alignment feature; the first sample mapping feature is spliced ​​after the second sample mapping feature to obtain the second sample alignment feature.

[0097] After obtaining the first sample alignment feature and the second sample alignment feature, the fusion alignment sub-loss is calculated using the target feature difference between the first sample alignment feature and the second sample alignment feature. Concatenate to the first sample mapping feature The tail of the first sample alignment feature is obtained ; Then map the first sample feature Splice to the second sample mapping feature The tail of the second sample alignment feature is obtained Finally, the above ring feature alignment loss function is used Formula (1) calculates the first sample alignment feature Align features with the second sample The fusion alignment sub-loss between them is:

[0098] (3)

[0099] Fusion Alignment Loss The first sample alignment feature can be aligned Align features with the second sample Constraints are imposed, and the integrity of the features can be fully considered during the fusion alignment process. If the distributions of two features are very similar, then no matter whether the second sample mapping feature Concatenate to the first sample mapping feature The tail or the first sample mapping feature Splice to the second sample mapping feature At the tail of , the fused features will still be distributed very similarly. Therefore, fusion alignment will ignore the local information of the features and focus on the overall expressiveness of the features.

[0100] In other embodiments, when the alignment module 230 runs the interpolation alignment mode, the training loss obtained by the target matching model 200 includes the interpolation alignment sub-loss. The interpolation alignment operation mode is: interpolating based on the first sample mapping feature and the second sample mapping feature in a second order to obtain the first sample alignment feature, and interpolating based on the first sample mapping feature and the second sample mapping feature in the reverse order of the second order to obtain the second sample alignment feature. The second order may be that the first sample mapping feature is before the second sample mapping feature. The interpolation alignment sub-loss is obtained by the target feature difference between the first sample alignment feature and the second sample alignment feature after the interpolation alignment process.

[0101] Specifically, the first sample mapping feature and the second sample mapping feature Cross-interpolate according to its channel dimension. For example, map the first sample to feature Interpolate the features in the second sample map Before, get the first sample alignment feature ; Map the second sample feature each time according to the channel dimension First sample mapping features Before, get the second sample alignment feature Finally, the above ring feature alignment loss function is used Formula (1) calculates the first sample alignment feature Align features with the second sample The interpolation alignment sub-loss between:

[0102] (4)

[0103] In the process of interpolation alignment, the locality of features can be fully considered. By aligning each feature channel, the distribution of features in the feature channel dimension is aligned. In other words, if the distribution of two features in the channel dimension is aligned, then no matter whether the first sample mapping feature Interpolate the features in the second sample map Before, it was the second sample mapping feature Interpolate the features in the first sample map Previously, the distribution of features after interpolation would still be very similar, otherwise they would be completely different. Therefore, the fusion alignment method will ignore the overall expressiveness of the features and focus on the local distribution of the features on the channel.

[0104] In some embodiments, in order to comprehensively align the first sample mapping feature and the second sample mapping feature from the three aspects of original features, overall feature expression, and local distribution of feature channels, so that the detection effect of the target matching model 200 is more obvious, the alignment module 230 can also simultaneously run the three modes of original alignment, fusion alignment, and interpolation alignment. Therefore, the training loss obtained by the target matching model 200 includes the three losses of original alignment sub-loss, fusion alignment sub-loss, and interpolation alignment sub-loss. Specifically, the training loss at this time Can be:

[0105] (5)

[0106] Step S340: Adjusting network parameters of the target matching model based at least on the training loss.

[0107] In some embodiments, the training loss can be directly used to adjust the network parameters of the target matching model 200 so that the detection effect of the target matching model 200 reaches the expected level. Figure 2 , using the adjustment module 240 in the target matching model 200 to adjust the network parameters of the target matching model 200 according to the training loss.

[0108] In other embodiments, after obtaining the training loss, the matching loss can also be obtained based on the matching degree between the first sample data feature and the second sample data feature and the sample label. The adjustment module 240 adjusts the network parameters of the target matching model 200 based on the training loss and the matching loss.

[0109] For example, the cosine similarity can be used to calculate the matching degree between the first sample data feature and the second sample data feature, and the matching degree obtained is compared with the matching threshold, and the result is output. If the matching degree is greater than the matching threshold, the output result is that the meanings expressed by the first sample data feature and the second sample data feature are the same; otherwise, the output result is different. The output result is then judged to be consistent with the sample label to determine the matching loss.

[0110] In addition, in the task of cross-text and visible light image target detection, the present application can directly introduce the idea of ​​feature alignment based on the general cross-text and visible light image target detection large model GLIP (Grounded Language-Image Pre-training, multimodal language image model) to constrain the feature distribution of the two modal data. In the existing GLIP algorithm, after encoding the text using the text encoder and encoding the visible light image using the image encoder, the encoded text and visible light image are split using the language image perception deep fusion module to obtain text words and image regions, and feature extraction is performed on the text words and image regions to obtain word features and region features, and the cosine similarity of the word features and region features is calculated. Afterwards, when matching the text words and image regions, the most similar text words and image regions are selected using the calculated similarity to determine the matching loss. In the present application, an alignment module is added to the GLIP algorithm. After the language image perception deep fusion module performs feature extraction on the encoded text and visible light image to obtain word features and region features, the alignment module maps the word features and region features to obtain first sample mapping features and second sample mapping features, and then performs original alignment, fusion alignment and interpolation alignment on the first sample mapping features and the second sample mapping features to calculate the training loss, and finally adjust the parameters in the GLIP algorithm based on the training loss and matching loss.

[0111] In a training scenario, the first sample data selected is picture data, which is an image of a puppy; the second sample data selected is text data, which is "a dog biting a frisbee", and the samples in the first sample data and the second sample data are annotated to have the same meaning.

[0112] The first sample data and the second sample data are input into the target matching model 200 for splitting, and then feature extraction is performed to obtain a plurality of first sample data features and second sample data features. Then, the first sample data features and the second sample data features are simultaneously subjected to three alignment methods: original alignment, fusion alignment, and interpolation alignment. Regarding the original alignment, the first sample data features are first mapped to obtain the first sample mapping features, and the second sample data features are mapped to obtain the second sample mapping features of the same dimension as the first sample mapping features, and then the first sample mapping features are directly selected as the first sample alignment features after the original alignment processing. , and directly select the second sample mapping feature as the second sample alignment feature after the original alignment processing , and then use the above formula (2) to calculate the original alignment loss Regarding fusion alignment, the first sample data feature is first mapped to obtain the first sample mapping feature, the second sample data feature is mapped to obtain the second sample mapping feature of the same dimension as the first sample mapping feature, and then the first sample mapping feature is concatenated before the second sample mapping feature to obtain the first sample alignment feature. ; After splicing the first sample mapping feature with the second sample mapping feature, the second sample alignment feature is obtained Finally, the above formula (3) is used to calculate the fusion alignment loss Regarding interpolation alignment, the first sample data feature is first mapped to obtain the first sample mapping feature, the second sample data feature is mapped to obtain the second sample mapping feature of the same dimension as the first sample mapping feature, and then the first sample mapping feature is interpolated before the second sample mapping feature to obtain the first sample alignment feature. ; Map the second sample feature before the first sample feature to obtain the second sample alignment feature Finally, the interpolation alignment loss is calculated using the above formula (4): .

[0113] Then the original alignment loss Fusion Alignment Loss and interpolation alignment sub-loss Sum to get the training loss . At the same time, the matching degree between the first sample data features and the second sample data features is calculated, and the matching degree is compared with the matching threshold, and the comparison result is output. If the matching degree is greater than the matching threshold, the comparison result is that the meanings expressed by the first sample data features and the second sample data features are the same; otherwise, the comparison result is different, so that the specific area corresponding to the phrase in the text can be accurately found in the picture. The comparison result is compared with the sample label to determine the matching loss. Finally, the network parameters of the target matching model 200 are adjusted in combination with the training loss and the matching loss.

[0114] This application proposes an explicit feature alignment idea and designs an alignment module based on this idea. It uses three alignment modes: original alignment, fusion alignment and interpolation alignment to strongly align text features and visible light visual features from the overall and channel distribution aspects of the features, reduce the difference between the two modalities, improve the accuracy of the cross-modal target matching model, and also enhance the performance of the cross-modal target detection algorithm.

[0115] And in order to better guide and constrain the alignment process of the alignment module during training, this application also proposes an adaptive circular feature alignment loss function to adaptively constrain the alignment process of features between the two modalities. This loss function is essentially a contrast loss function, which is roughly distributed in a ring. It ignores samples with extremely small distribution differences between the two modal features (approximately no modal difference), thereby avoiding over-learning of samples with extremely small distribution differences between the two modal features. At the same time, samples with extremely large differences in the distribution of the two modal features are also ignored in the current training to avoid forced learning of samples with extremely large differences in the distribution of the two modal features. Instead, as the training progresses, the coupling of the two modalities becomes stronger and stronger, and the samples with extremely large distribution differences become learnable samples, and then the contrast loss is used for constraint, step by step.

[0116] Therefore, the target matching model in this application can be used in any task that requires feature alignment, such as multimodal classification, multimodal retrieval, multimodal segmentation, etc.

[0117] See also Figure 4 , Figure 4 It is a flow chart of an embodiment of the target detection network of the present application. The target detection network 400 includes an acquisition module 410, a feature extraction module 420, an alignment fusion module 430 and a matching module 440. The acquisition module 410 executes the acquisition of the first target data and the second target data of different modalities, wherein the modalities of the first target data and the second target data are any two of image, text, audio and video. The feature extraction module 420 executes the extraction of the first target feature corresponding to the first sub-target data in the first target data, and extracts the second target feature corresponding to the second sub-target data in the second target data. The alignment fusion module 430 executes the feature alignment result based on the first target feature and the second target feature to obtain the first similarity, and obtains the second similarity based on the feature fusion result between the first target feature and the second target feature. The matching module 440 executes the fusion similarity based on the first similarity and the second similarity to obtain a matching result that characterizes whether the first sub-target data and the second sub-target data have the same meaning.

[0118] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0119] See also Figure 5 , Figure 5: is a schematic diagram of a framework of an embodiment of an electronic device 50 of the present application. The electronic device 50 includes a memory 51 and a processor 52 coupled to each other, and the processor 52 is used to execute program instructions stored in the memory 51 to implement the steps of any of the above-mentioned target matching model training method embodiments, or to implement the steps of any of the above-mentioned target matching method embodiments. In a specific implementation scenario, the electronic device 50 may include, but is not limited to: a microcomputer, a server, and in addition, the electronic device 50 may also include mobile devices such as laptops and tablet computers, which are not limited here.

[0120] Specifically, the processor 52 is used to control itself and the memory 51 to implement the steps of the training method embodiment of any of the above-mentioned target matching models, or to implement the steps of any of the above-mentioned target matching method embodiments. The processor 52 can also be called a CPU (Central Processing Unit). The processor 52 may be an integrated circuit chip with signal processing capabilities. The processor 52 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 52 can be implemented by an integrated circuit chip.

[0121] See also Figure 6 , Figure 6 1 is a schematic diagram of a framework of an embodiment of a computer-readable storage medium 60 of the present application. The computer-readable storage medium 60 stores program instructions 601 that can be executed by a processor, and the program instructions 601 are used to implement the steps of any of the above-mentioned target matching model training method embodiments, or to implement the steps of any of the above-mentioned target matching method embodiments.

[0122] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0123] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0124] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0125] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0126] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

Claims

1. A target matching method, It is characterized in that include: Acquire first target data and second target data in different modalities, wherein the modalities of the first target data and the second target data are any two of image, text, audio, and video; Extracting a first target feature corresponding to the first sub-target data in the first target data, and extracting a second target feature corresponding to the second sub-target data in the second target data; Based on the first target feature, mapping obtains a first mapping feature, and based on the second target feature, mapping obtains a second mapping feature of the same dimension as the first mapping feature; Obtaining a first similarity based on a feature alignment result between the first mapping feature and the second mapping feature, and obtaining a second similarity based on a feature fusion result between the first target feature and the second target feature; Based on the fused similarity of the first similarity and the second similarity, a matching result is obtained, indicating whether the first sub-target data and the second sub-target data have the same meaning.

2. The method according to claim 1, It is characterized in that The obtaining a first similarity based on a feature alignment result between the first mapping feature and the second mapping feature includes: Acquire a first alignment feature and a second alignment feature after the first mapping feature and the second mapping feature are processed by the same alignment method respectively; The first similarity is obtained based on the first alignment feature and the second alignment feature.

3. The method according to claim 2, It is characterized in that The first similarity includes at least one of original alignment similarity, fused alignment similarity, and interpolated alignment similarity; The original alignment similarity is measured when the alignment mode is original alignment, the fused alignment similarity is measured when the alignment mode is fused alignment, and the interpolated alignment similarity is measured when the alignment mode is interpolated alignment.

4. The method according to claim 2, It is characterized in that The first similarity at least includes the original alignment similarity, the alignment method at least includes the original alignment, and the obtaining of the first alignment feature and the second alignment feature after the first mapping feature and the second mapping feature are processed by the same alignment method respectively includes: The first mapping feature is directly selected as the first alignment feature after the original alignment process, and the second mapping feature is directly selected as the second alignment feature after the original alignment process.

5. The method according to claim 2, It is characterized in that The first similarity at least includes fusion alignment similarity, the alignment method at least includes fusion alignment, and the obtaining of the first alignment feature and the second alignment feature after the first mapping feature and the second mapping feature are processed by the same alignment method respectively includes: The first alignment feature is obtained by fusing the first mapping feature and the second mapping feature in a first order, and the second alignment feature is obtained by fusing the first mapping feature and the second mapping feature in a reverse order of the first order.

6. The method according to claim 5, It is characterized in that The fusing the first mapping feature and the second mapping feature in a first order to obtain the first alignment feature includes: splicing the first mapping feature before the second mapping feature to obtain the first alignment feature; The fusing the first mapping feature and the second mapping feature in the reverse order of the first order to obtain the second alignment feature includes: The first mapping feature is concatenated with the second mapping feature to obtain the second alignment feature.

7. The method according to claim 2, It is characterized in that The first similarity at least includes interpolation alignment similarity, the alignment method at least includes interpolation alignment, and the obtaining of the first alignment feature and the second alignment feature after the first mapping feature and the second mapping feature are processed by the same alignment method respectively includes: The first alignment feature is obtained by interpolating the first mapping feature and the second mapping feature in a second order, and the second alignment feature is obtained by interpolating the first mapping feature and the second mapping feature in a reverse order of the second order.

8. The method according to claim 1, It is characterized in that The matching result is predicted by a target matching model, and the training method of the target matching model includes: Extracting first sample data features of first sample data, and extracting second sample data features of second sample data, wherein the modalities of the first sample data and the second sample data are any two of image, text, audio, and video; Based on the first sample data feature, mapping obtains a first sample mapping feature, and based on the second sample data feature, mapping obtains a second sample mapping feature of the same dimension as the first sample mapping feature; Obtaining a training loss based on a magnitude relationship of a target feature difference between the first sample mapping feature and the second sample mapping feature relative to a target difference range; Based at least on the training loss, a network parameter of the target matching model is adjusted.

9. The method according to claim 8, It is characterized in that Before obtaining the training loss based on the magnitude relationship of the target feature difference between the first sample mapping feature and the second sample mapping feature relative to the target difference range, the method further includes: Obtaining a first sample alignment feature and a second sample alignment feature after the first sample mapping feature and the second sample mapping feature are processed by the same alignment method respectively; Using a plurality of measurement methods to measure the difference between the first sample alignment feature and the second sample alignment feature respectively, to obtain sub-feature differences of various measurement methods, wherein the plurality of measurement methods at least include: Euclidean distance and cosine distance; Based on the sub-feature differences of various measurement methods, the target feature difference is obtained.

10. The method according to claim 9, It is characterized in that The alignment method includes original alignment, the training loss includes an original alignment sub-loss, and the original alignment sub-loss is obtained by a target feature difference between a first sample alignment feature and a second sample alignment feature after the original alignment process.

11. The method according to claim 9, It is characterized in that The alignment method includes fusion alignment, the training loss includes a fusion alignment sub-loss, and the fusion alignment sub-loss is obtained by a target feature difference between a first sample alignment feature and a second sample alignment feature after the fusion alignment process.

12. The method according to claim 9, It is characterized in that The alignment method includes interpolation alignment, the training loss includes an interpolation alignment sub-loss, and the interpolation alignment sub-loss is obtained by a target feature difference between a first sample alignment feature and a second sample alignment feature after the interpolation alignment process.

13. The method according to claim 9, It is characterized in that The obtaining of the training loss based on the magnitude relationship of the target feature difference between the first sample mapping feature and the second sample mapping feature relative to the target difference range includes: In response to the target feature difference being within the target difference range, a sum of sub-feature differences of various measurement methods is obtained as a fused feature difference, and a weight parameter is obtained with the target feature difference as a base and a preset hyperparameter as an exponent; The training loss is obtained based on the fused feature difference and the weight parameter.

14. An electronic device, It is characterized in that It comprises a memory and a processor coupled to each other, wherein the processor is used to execute program instructions stored in the memory to implement the target matching method according to any one of claims 1 to 13.

15. A computer-readable storage medium having program instructions stored thereon, It is characterized in that When the program instructions are executed by a processor, the target matching method according to any one of claims 1 to 13 is implemented.

Citation Information

Patent Citations

  • Text image matching method and device thereof, computer equipment and storage medium

    CN113516143A

  • Disease diagnosis method and device based on multi-modal data, equipment and medium

    CN116259407A