A target recognition method, device and electronic equipment

Abstract

This application discloses a method, apparatus, and electronic device for target recognition. The method includes extracting multiple features of different modalities of a target object from a video to be processed, and determining reference features corresponding to the video to be processed. Here, the reference features can be determined based on features from multiple reference videos, and the reference videos are videos with features of at least one modality among different modalities. Then, based on the determined reference features, the multiple features of the video to be processed are fused to obtain fused features of the video to be processed. Finally, the fused features are used to determine the recognition result of the target object. Based on the above method, fused features of different modalities of the target object can be obtained, solving the problem of low target recognition accuracy caused by single modal features in the prior art. Furthermore, combining reference features to fuse the extracted features of different modalities can effectively improve the accuracy of target recognition.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method, apparatus and electronic device for target recognition. Background Technology

[0002] Currently, in various identity recognition scenarios such as urban security systems and company attendance systems, facial recognition is generally used to extract the facial features of the target object in the collected video images. By recognizing the facial features, the identity information of the target object can be determined.

[0003] However, when the target object in the video image is occluded, the above method is usually not able to identify the target object in the video image. Therefore, the existing technology has the problem of low accuracy in identifying target objects in video images. Summary of the Invention

[0004] This application provides a method, apparatus, and electronic device for target recognition, which combines reference features to fuse extracted modal features to obtain fused features of the video to be processed, and performs target recognition based on the fused features, effectively improving the accuracy of target recognition and solving the problem of low target recognition accuracy caused by single modal features in the prior art.

[0005] Firstly, this application provides a method for target recognition, the method comprising:

[0006] Extract multiple features of different modalities of the target object in the video to be processed;

[0007] Determine reference features corresponding to the video to be processed; wherein, the reference features are determined based on features of multiple reference videos, and the reference videos are videos having features of at least one of the different modalities;

[0008] Based on the reference features, the multiple features of the video to be processed are fused to obtain the fused features of the video to be processed;

[0009] The identification result of the target object is determined using the fusion features.

[0010] By combining the above method with reference features to fuse the extracted modal features, the fused features of the video to be processed are obtained, and target recognition is performed based on the fused features. This solves the problem of low target recognition accuracy caused by single modal features in existing technologies. This method can not only effectively improve the quality of fused features, but also effectively improve the accuracy of target recognition based on the fused features.

[0011] In one possible design, the extraction of multiple features of different modalities of the target object in the video to be processed includes:

[0012] Extract a first set of images from the video to be processed, and perform the following operations on each image in the first set of images:

[0013] Calculate the similarity value between a single image and each image in the first image set. If all similarity values ​​are greater than a preset similarity threshold, then add the single image to the second image set.

[0014] In the second image set, features of different modalities of the target object are extracted from each image;

[0015] Weighted summation is performed on multiple features of the same modality extracted from each image to calculate one feature of the same modality, thereby obtaining multiple features of different modalities;

[0016] The calculated features of the different modalities are used as the features of the different modalities of the video to be processed.

[0017] Based on the above method, a method for extracting multiple features of different modalities from a video to be processed is proposed. For each image in the first image set of the video to be processed with a similarity value greater than a preset similarity threshold, features of multiple modalities in each image are extracted. The extracted features of the same modality are then weighted and summed, so that one modality corresponds to one weighted feature, thus obtaining multiple features of different modalities in the video to be processed. This method can effectively save computational costs and help improve the quality of the fused features generated from the video to be processed.

[0018] In one possible design, extracting the first image set from the video to be processed includes:

[0019] Extract multiple images from the video to be processed as a third image set, and calculate the image quality score of each image in the third image set;

[0020] Extract all images with quality scores greater than a preset threshold to form a first image set.

[0021] The above method can effectively filter out images with high image quality scores from the video to be processed, reduce input noise, and further reduce the negative impact of low image quality scores on inaccurate feature extraction, feature fusion, and target recognition in subsequent processes.

[0022] In one possible design, after extracting multiple features of different modalities of the target object in the video to be processed, the following is also included:

[0023] The video to be processed is feature-encoded to obtain the video features corresponding to the video to be processed.

[0024] The video features are added one by one to each of the multiple features to obtain the encoded features corresponding to each feature, and the obtained multiple encoded features are used as multiple features of the video to be processed.

[0025] By using the above method, cross-modal encoding of multiple features extracted from different modalities in the video to be processed can be performed to complete the information interaction between features of different modalities in the video to be processed. Based on the encoded features of different modalities, the quality of fused features and the accuracy of target recognition can be effectively improved.

[0026] In one possible design, determining the reference features corresponding to the video to be processed includes:

[0027] Calculate the similarity value between each of the multiple features and each feature in the preset video to obtain multiple similarity values ​​for each of the multiple features;

[0028] According to the magnitude of the similarity value, the multiple similarity values ​​of each feature among the multiple features are arranged, and the preset video corresponding to the similarity value arranged at the target position is taken as the reference video;

[0029] Features of different modalities are extracted from each reference video, and the extracted features are used as reference features for the video to be processed.

[0030] Using the above method, reference features corresponding to the video to be processed can be determined. For each modality of the video to be processed, the similarity value between the feature and the feature of the same modality in the reference video is calculated. This is used to select reference videos, and the features of the reference videos are used as reference features of the video to be processed. These reference features can be used to improve the quality of multiple features of different modalities of the video to be processed, and feature fusion is performed. This improves the accuracy of target recognition based on the fused features.

[0031] In one possible design, the step of extracting features of the different modalities from each reference video and using the extracted features as reference features for the video to be processed includes:

[0032] Determine whether each reference video contains missing features of the different modalities;

[0033] If not, then extract the features of the different modalities in each of the reference videos, and use the extracted features as the reference features corresponding to the video to be processed;

[0034] If so, then extract the features of the different modalities in each of the reference videos, fill the extracted missing features with a specified vector, and use the filled features of the different modalities as the reference features corresponding to the video to be processed.

[0035] Based on the above method, a method for filling in the features of different modalities of a reference video is proposed. That is, when there are features in the reference video that do not have all the features of different modalities of the video to be processed, the missing features are taken as missing features and a specified vector is used to fill the missing features. This can avoid the negative impact of the missing features on feature fusion, effectively improve the quality of the obtained fused features, and improve the recognition effect of feature recognition based on fused features.

[0036] In one possible design, fusing the multiple features of the video to be processed based on the reference features to obtain the fused features of the video to be processed includes:

[0037] Determine a feature matrix composed of the plurality of features and the reference features;

[0038] Obtain the adjacency matrix corresponding to the feature matrix; wherein, the adjacency matrix represents the connection relationship between different features in the feature matrix for fusion;

[0039] By aggregating the feature matrix and the adjacency matrix, the fused features of the video to be processed are obtained.

[0040] Based on the above method, a method is proposed to fuse multiple features of different modalities in the video to be processed based on feature matrix and adjacency matrix. Because the constructed feature matrix and the obtained adjacency matrix not only consider the information between different modalities in the video to be processed, but also consider the information of the same modalities between the video to be processed and the reference video, it can effectively improve the quality of the fused features of the video to be processed, and effectively improve the recognition accuracy of target recognition based on the fused features, and further reduce the false alarm rate of target recognition.

[0041] In one possible design, obtaining the adjacency matrix corresponding to the feature matrix includes:

[0042] Determine the connection coefficients between each feature in the plurality of features and each feature in the feature matrix for fusion;

[0043] Based on the determined connection coefficients, an adjacency matrix composed of the determined connection coefficients is obtained.

[0044] The above method is used to obtain the adjacency matrix. This adjacency matrix not only considers the feature information between different modalities, but also the information between the features of the same modality between the video to be processed and the reference video. That is, the adjacency matrix can improve the discriminability and robustness of the fused features, and further improve the recognition accuracy of target recognition based on fused features, and reduce the false alarm rate of target recognition based on fused features.

[0045] In one possible design, obtaining the fused features of the video to be processed by aggregating the feature matrix and the adjacency matrix includes:

[0046] In response to the absence of missing reference features in the feature matrix, a preset number of updates is obtained;

[0047] By using a graph neural network, the feature matrix and the adjacency matrix are aggregated after the preset number of updates to obtain the target feature matrix after the feature matrix is ​​updated; wherein, the target feature matrix is ​​composed of target features;

[0048] In the target feature matrix, multiple target features corresponding to the multiple features are extracted, and the multiple target features are fused to obtain the fused features of the video to be processed.

[0049] By employing the above method, a graph neural network is used to fuse features from different modalities of the video to be processed. Compared with existing technologies, this method can automatically learn the aggregation method of features from each modality. Furthermore, by combining the adjacency matrix, i.e., the reference features, the learned fused features become more discriminative and robust, effectively improving the quality of the fused features and further enhancing the recognition rate and accuracy of target recognition results based on these fused features.

[0050] In one possible design, obtaining the fused features of the video to be processed by aggregating the feature matrix and the adjacency matrix includes:

[0051] In response to the presence of a missing reference feature in the feature matrix, the connection coefficients associated with the missing reference feature in the adjacency matrix are adjusted to a specified value;

[0052] By using a graph neural network, the feature matrix and the adjusted adjacency matrix are aggregated after a preset number of updates to obtain the target feature matrix after the feature matrix is ​​updated.

[0053] The target feature matrix is ​​updated again based on the preset mask matrix and the preset scaling matrix to obtain the fused features of the video to be processed.

[0054] Based on the above method, a fusion method for feature loss is proposed. By modifying the corresponding adjacency matrix, the influence of missing modality feature information on the generated target feature matrix is ​​prevented. Furthermore, the target feature matrix is ​​updated again by using a preset mask matrix and a preset scaling matrix. This effectively solves the problem of modality feature loss in practical application scenarios, effectively improves the quality of the obtained fused features, and increases the accuracy of target recognition based on fused features.

[0055] Secondly, this application provides a target recognition device, the device comprising:

[0056] The extraction module extracts multiple features of different modalities of the target object in the video to be processed;

[0057] The determination module determines the reference features corresponding to the video to be processed; wherein, the reference features are determined based on the features of multiple reference videos, and the reference videos are features of videos having features of at least one of the different modalities;

[0058] The fusion module, based on the reference features, fuses the multiple features of the video to be processed to obtain the fused features of the video to be processed;

[0059] The identification module uses the fused features to determine the identification result of the target object.

[0060] In one possible design, the extraction module is specifically used for:

[0061] Extract a first set of images from the video to be processed, and perform the following operations on each image in the first set of images:

[0062] Calculate the similarity value between a single image and each image in the first image set. If all similarity values ​​are greater than a preset similarity threshold, then add the single image to the second image set.

[0063] In the second image set, features of different modalities of the target object are extracted from each image;

[0064] Weighted summation is performed on multiple features of the same modality extracted from each image to calculate one feature of the same modality, thereby obtaining multiple features of different modalities;

[0065] The calculated features of the different modalities are used as the features of the different modalities of the video to be processed.

[0066] In one possible design, the extraction module is specifically used for:

[0067] Extract multiple images from the video to be processed as a third image set, and calculate the image quality score of each image in the third image set;

[0068] Extract all images with quality scores greater than a preset threshold to form a first image set.

[0069] In one possible design, the extraction module is further configured to perform feature encoding on the video to be processed to obtain video features corresponding to the video to be processed;

[0070] The video features are added one by one to each of the multiple features to obtain the encoded features corresponding to each feature, and the obtained multiple encoded features are used as multiple features of the video to be processed.

[0071] In one possible design, the determining module is specifically used for:

[0072] Calculate the similarity value between each of the multiple features and each feature in the preset video to obtain multiple similarity values ​​for each of the multiple features;

[0073] According to the magnitude of the similarity value, the multiple similarity values ​​of each feature among the multiple features are arranged, and the preset video corresponding to the similarity value arranged at the target position is taken as the reference video;

[0074] Features of different modalities are extracted from each reference video, and the extracted features are used as reference features for the video to be processed.

[0075] In one possible design, the determining module is specifically used for:

[0076] Determine whether each reference video contains missing features of the different modalities;

[0077] If not, then extract the features of the different modalities in each of the reference videos, and use the extracted features as the reference features corresponding to the video to be processed;

[0078] If so, then extract the features of the different modalities in each of the reference videos, fill the extracted missing features with a specified vector, and use the filled features of the different modalities as the reference features corresponding to the video to be processed.

[0079] In one possible design, the fusion module is specifically used for:

[0080] Determine a feature matrix composed of the plurality of features and the reference features;

[0081] Obtain the adjacency matrix corresponding to the feature matrix; wherein, the adjacency matrix represents the connection relationship between different features in the feature matrix for fusion;

[0082] By aggregating the feature matrix and the adjacency matrix, the fused features of the video to be processed are obtained.

[0083] In one possible design, the fusion module is specifically used for:

[0084] Determine the connection coefficients between each feature in the plurality of features and each feature in the feature matrix for fusion;

[0085] Based on the determined connection coefficients, an adjacency matrix composed of the determined connection coefficients is obtained.

[0086] In one possible design, the fusion module is specifically used for:

[0087] In response to the absence of missing reference features in the feature matrix, a preset number of updates is obtained;

[0088] By using a graph neural network, the feature matrix and the adjacency matrix are aggregated after the preset number of updates to obtain the target feature matrix after the feature matrix is ​​updated; wherein, the target feature matrix is ​​composed of target features;

[0089] In the target feature matrix, multiple target features corresponding to the multiple features are extracted, and the multiple target features are fused to obtain the fused features of the video to be processed.

[0090] In one possible design, the fusion module is specifically used for:

[0091] In response to the presence of a missing reference feature in the feature matrix, the connection coefficients associated with the missing reference feature in the adjacency matrix are adjusted to a specified value;

[0092] By using a graph neural network, the feature matrix and the adjusted adjacency matrix are aggregated after a preset number of updates to obtain the target feature matrix after the feature matrix is ​​updated.

[0093] The target feature matrix is ​​updated again based on the preset mask matrix and the preset scaling matrix to obtain the fused features of the video to be processed.

[0094] Thirdly, this application provides an electronic device, the electronic device comprising:

[0095] Memory, used to store computer programs;

[0096] When a processor executes a computer program stored in the memory, it implements the above-described method steps for target recognition.

[0097] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method steps for target recognition.

[0098] For the various aspects of the second to fourth aspects mentioned above, and the technical effects that each aspect may achieve, please refer to the above description of the technical effects that can be achieved for the first aspect or the various possible solutions in the first aspect, which will not be repeated here. Attached Figure Description

[0099] Figure 1 A flowchart of a target recognition method provided in this application;

[0100] Figure 2 A schematic diagram of a facial modality provided in this application;

[0101] Figure 3 A schematic diagram of a human body modality provided in this application;

[0102] Figure 4 A schematic diagram illustrating a single-modal multi-feature weighted fusion method provided in this application;

[0103] Figure 5 A schematic diagram of a cross-modal coding method provided in this application;

[0104] Figure 6 A schematic diagram of a target recognition device provided in this application;

[0105] Figure 7 A schematic diagram of the structure of an electronic device provided in this application. Detailed Implementation

[0106] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The specific operational methods in the method embodiments can also be applied to the device embodiments or system embodiments. It should be noted that in the description of this application, "multiple" is understood as "at least two". "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. A connected to B can represent: A and B directly connected, and A and B connected through C. Furthermore, in the description of this application, terms such as "first" and "second" are used only for distinguishing the purpose of description and should not be construed as indicating or implying relative importance or order.

[0107] This application provides a method, apparatus, and electronic device for target recognition, which obtains fused features of different modalities of a target object, solving the problem of low target recognition accuracy caused by single modal features in the prior art. Furthermore, by combining reference features to fuse the extracted features of different modalities, the accuracy of target recognition can be effectively improved.

[0108] The method provided in the embodiments of this application will be further described in detail below with reference to the accompanying drawings.

[0109] See Figure 1 As shown in the figure, this application provides a method for target recognition, and the specific process is as follows:

[0110] Step 101: Extract multiple features of different modalities of the target object in the video to be processed;

[0111] In the application embodiment, multiple images from the video to be processed are first extracted as a third image set. Then, the image quality score of each image in the third image set is calculated, and all images with image quality scores greater than a preset threshold are extracted to form a first image set.

[0112] Specifically, the method for extracting multiple images from the video to be processed as a third image set can be as follows: extract multiple images from the video to be processed at equal intervals, and use the extracted multiple images as images in the third image set. Here, the equal interval can be a preset time period or a preset number of images.

[0113] The aforementioned image quality score can be used to characterize the sharpness of an image and / or the ratio of the target object occlusion area to the target object area in the image. That is, the sharper the image and / or the lower the proportion of the target object occlusion area in the image, the higher the calculated image quality score; the blurrier the image and / or the higher the proportion of the target object occlusion area in the image, the lower the calculated image quality score.

[0114] Here, when the target object is a person, the image quality score of each image may include, but is not limited to: the face quality score of the face image in each image, and the human body quality score of the human body image in each image.

[0115] Taking face quality score as an example, the target object region can be the area where the target object's face is located in the image. If the face is clearer and / or more complete, the face quality score calculated for the image is higher; if the face is blurrier, the face has a larger skew angle, and / or the face is more obscured, the face quality score calculated for the image is lower.

[0116] After extracting the first set of images from the video to be processed using the above method, the following operations can be performed on each image in the first set of images:

[0117] Calculate the similarity value between a single image and each image in the first image set. If all similarity values ​​are greater than a preset similarity threshold, then add the single image to the second image set.

[0118] For example, if the first image set contains three images: image A, image B, and image C, taking the above operation on image A as an example, calculate the first similarity value between image A and image B respectively; calculate the second similarity value between image A and image C. If both the first similarity value and the second similarity value are greater than the preset similarity threshold, then add image A to the second image set.

[0119] It is worth noting that the preset similarity threshold can be determined based on the actual application.

[0120] After performing the above operations on each image in the first image set, the features of different modalities of the target object in each image can be determined in the generated second image set. Then, the multiple features of the same modality extracted from each image are weighted and summed to calculate one feature corresponding to the same modality. This process is repeated for multiple features corresponding to different modalities, and the calculated multiple features corresponding to different modalities are used as multiple features of different modalities in the video to be processed.

[0121] Specifically, different modalities represent two or more modalities of the target object. In this application embodiment, different modalities can be divided based on the characteristics of different forms of the target object. For example, when the target object is a person, information such as the person's voice, limbs / movements, gait, and clothing can be used as a modal. In this application embodiment, different modalities can also be divided based on the characteristics of different components / parts of the target object. For example, when the target object is a person, information such as the person's face, hands, body, head and shoulders, and legs can be used as a modal. Moreover, the different modalities in this application embodiment can include any number of modalities divided according to different methods. For example, when the target object is a person, the modalities in this application embodiment can include the target object's face, head, body, voice, etc.

[0122] Here, corresponding modal features can be extracted for each modality, such as facial features, head features, body features, and voice features of the target object.

[0123] like Figure 2 As shown, the detection part 1 of the target object is detected, and the features of the detection part 1 are extracted as the features of the target object's face modality.

[0124] like Figure 3 As shown, the detection part 2 of the target object is detected, and the features of the detection part 2 are extracted as the features of the target object's body modality.

[0125] For example, suppose the second set contains three images: image D, image E, and image F, and extract features from three different modalities in the above three images: the first modality is the face, the second modality is the body, and the third modality is speech.

[0126] like Figure 4 As shown, taking the first modality as an example, the first modality features D of the face in image D, the first modality features E of the face in image E, and the first modality features F of the face in image F are extracted respectively. Then, by weighted fusion of the first modality features D, E, and F, the weighted fused face modality features of images D, E, and F are obtained. Here, the face modality features are the features corresponding to the face in the video to be processed.

[0127] In addition, the weight coefficients of each modal feature in the above weighted fusion can be the normalized image quality scores of each image. Of course, these weight coefficients can also be determined according to the actual application, which will not be elaborated on here.

[0128] Using the above method, multiple features of different modalities of the target object in the video to be processed can be determined.

[0129] Furthermore, in one possible scenario, if a missing modality exists among multiple features of different modalities in the video to be processed, the missing modality is replaced with a specified vector.

[0130] For example, when extracting three features from the target object in a video of the target object—face, body, and speech—but the above calculations reveal that the speech modality of the target object in the video of the target object is missing, then by representing the missing modality with a specified vector, the three features of the target object in the video of the target object—face, body, and speech—are obtained.

[0131] It is worth noting that the specified vector mentioned above is usually the zero vector. Of course, other specified vectors can be determined according to the actual application, which will not be elaborated here.

[0132] Furthermore, in one possible design, cross-modal feature encoding will be performed on the features of different modalities in the video to be processed. Specifically, the video to be processed is first encoded to obtain the video features corresponding to the video to be processed. Then, the video features are added one by one to each of the multiple features of the target object in different modalities in the video to be processed to obtain the encoded features corresponding to each feature. The multiple encoded features obtained are then used as multiple features of the video to be processed.

[0133] Specifically, the detailed calculation formula for adding video features one by one to each of the multiple features of the target object in different modalities in the video to be processed can be found in Formula 1 below.

[0134] h m =g(f video f m ) = ReLU(W m (f m ||f video ) (Formula 1)

[0135] Among them, h m f represents the features of a single modality in the encoded video to be processed. video f represents the video features of the video to be processed. m W represents the features of a single modality in the video to be processed before encoding. m This is a preset vector.

[0136] It is worth noting that the above h m ∈R d The above f video ∈R d The above The above d represents the dimension of the feature, which is used here to ensure that the dimension of each feature participating in cross-modal coding is consistent, and m represents the modality corresponding to the feature.

[0137] For example, such as Figure 5 As shown, if three features of three different modalities—face, body, and speech—are determined in the video to be processed, then the video features of the video to be processed are calculated. Then, using the calculation method of Formula 1 above, the encoded features of the face modality of the video features added to the face modality, the encoded features of the body modality of the video features added to the body modality, and the encoded features of the speech modality of the video features added to the speech modality are calculated respectively.

[0138] Here, the video features of the video to be processed can be represented as f videoThe facial features of the video to be processed can be represented as f1, the human body features of the video to be processed can be represented as f2, and the speech features of the video to be processed can be represented as f3. Based on formula 3, the encoded facial features h1, the encoded human body features h2, and the encoded speech features h3 can be obtained.

[0139] The above method can solve the problem of poor fusion feature quality caused by the heterogeneity of information between features of different modalities, and can complete the information interaction between multiple features of different modalities in the video to be processed. In this way, it is beneficial to improve the accuracy of fusion features and the recognition accuracy of target recognition based on fusion features.

[0140] In summary, this step can extract multiple features of different modalities of the target object in the video to be processed.

[0141] Step 102: Determine the reference features corresponding to the video to be processed;

[0142] In this embodiment, after extracting multiple features of different modalities of the target object in the video to be processed, the similarity value between each of these multiple features and each feature in the preset video is calculated to obtain multiple similarity values ​​for each of these multiple features. The multiple similarity values ​​of each of these multiple features are arranged according to the magnitude of the calculated similarity values. Then, the preset video corresponding to the similarity value at the target position is taken as the reference video. Finally, the features of different modalities in each reference video are extracted, and the extracted features are used as the reference features corresponding to the video to be processed.

[0143] Here, the aforementioned reference features are features determined based on features from multiple reference videos, where each reference video is a video having features from at least one of the different modalities of the video to be processed.

[0144] Specifically, after extracting multiple features of different modalities of the target object in the video to be processed through step 101, a preset video that meets the requirements is selected from the preset database based on the extracted different modalities of the target object.

[0145] For example, if we extract three features from the target object in the video to be processed—face, body, and voice—then we extract preset videos with features of any one or more of these three modalities from the preset database.

[0146] For example, if preset video 1 has features of one modality (face), preset video 2 has features of two modalities (face and body), preset video 3 has features of four modalities (face, body, voice, and head), and preset video 4 has features of one modality (head), then preset video 1, preset video 2, and preset video 3 can be extracted.

[0147] After determining the preset video, the similarity value between each feature of the video to be processed and each feature of the preset video is calculated, thus obtaining multiple similarity values ​​for each feature of the video to be processed.

[0148] Then, according to the calculated similarity values, the multiple similarity values ​​of each feature are arranged. The specific arrangement method can be to sort the similarity values ​​from smallest to largest or from largest to smallest. The preset video corresponding to the similarity value arranged at the target position is taken as the reference video.

[0149] It is worth noting that the target location mentioned above can be a pre-set location, a location determined by a preset threshold, or a location determined based on the actual application.

[0150] Specifically, reference videos can be selected using a K-nearest neighbor method. The specific selection method is as follows: for each modality of multiple features of the video to be processed, a search is performed, and the intersection of the first K1 results is taken as the K-nearest neighbor videos of the video to be processed. The K-nearest neighbor videos are the K reference videos.

[0151] Here, K1 can be a fixed value, representing one possible target location. The number of reference videos K determined by searching different modalities is uncertain and needs to be determined based on the actual intersection results.

[0152] Furthermore, to facilitate understanding of the intersection of the above results by those skilled in the art, taking three modal features A{2,3,4} and three modal features B{3,4,5} as an example, where each number is used to identify the object to which this modal feature belongs, the intersection of the results is {3,4}, that is, the intersection of the results is: modal feature A{3,4}, modal feature B{3,4}.

[0153] It is worth noting that the above-mentioned method of determining the reference video using K-nearest neighbors is only one possible way to determine the reference video. Other methods can also be used to determine the reference video. The purpose of determining the reference video is to extract reference features from the reference video. Based on the reference features, multiple features of the video to be processed are fused to make the fused features more discriminative and robust, thereby improving the quality of the fused features.

[0154] Furthermore, in one possible design, the reference videos determined by the above method may have missing modalities. In this case, it is necessary to extract features from multiple different modalities for each reference video, and these multiple modalities should correspond to the different modalities to which the various features of the image to be processed belong.

[0155] For example, if the three features of the image to be processed belong to three different modalities, namely face, body and speech, then it is necessary to extract the features of face, body and speech modalities for each reference video. If the features of face, body or speech modalities of the reference video cannot be extracted, then the unextractable modalities are regarded as the missing modalities of the reference video, that is, the reference video has a missing modality.

[0156] To address the aforementioned issues, in this embodiment of the application, the reference features corresponding to the video to be processed are determined by judging whether each reference video contains missing features of different modalities.

[0157] Specifically, the first step is to determine whether each reference video contains missing features of the different modalities.

[0158] If the reference videos do not have the missing features of the different modalities, then the features of the different modalities in each reference video are extracted, and the extracted features are used as the reference features corresponding to the video to be processed.

[0159] If each reference video has missing features of the different modalities, then the features of the different modalities in each reference video are extracted, and the extracted missing features are filled with a specified vector. The filled features of the different modalities are then used as the reference features corresponding to the video to be processed.

[0160] The reference features corresponding to the video to be processed are determined using the above method.

[0161] Step 103: Based on the reference features, fuse the multiple features of the video to be processed to obtain the fused features of the video to be processed;

[0162] In this embodiment of the application, a feature matrix composed of multiple features of the video to be processed and the reference features is determined based on the reference features corresponding to the video to be processed. Then, the adjacency matrix corresponding to the feature matrix is ​​obtained. Finally, the fused features of the video to be processed are obtained by aggregating the feature matrix and the adjacency matrix.

[0163] Here, the adjacency matrix can be used to represent the connection relationships between different features in the feature matrix.

[0164] Specifically, the adjacency matrix can be obtained as follows: determine the connection coefficients between each feature in the multiple features of the video to be processed and each feature in the feature matrix, and then obtain the adjacency matrix composed of the determined connection coefficients.

[0165] For example, multiple features of the video to be processed and a single feature from all reference features can be considered as nodes. These nodes together form a graph, where an edge connects two nodes. The graph can be divided into two types: modal edges connecting different modalities and nearest-neighbor edges connecting the video to be processed and reference videos that belong to the same modality. Here, modal edges can fuse feature information from different modalities, while nearest-neighbor edges can fuse reference feature (neighbor information) information of the target object. By connecting and aggregating nodes in the graph using these two types of edges, the goal of fusing multimodal features and fusing reference features can be achieved.

[0166] Referring to Formula 2, which is a method for calculating the weight of an edge in constructing an adjacency matrix provided in this application embodiment, the parameters involved in Formula 2 are explained below using a diagram as an example.

[0167]

[0168] Among them, A ij for h i and h j The weight h of the edges connecting them. i and h j Let i and j be the features of the i-th and j-th nodes that are connected in the graph.

[0169] Based on the calculation using Formula 2 above, we can obtain the adjacency matrix A, which is composed of the weights of the edges, and A∈R. n×n , where n is the number of nodes.

[0170] After determining the adjacency matrix, it is necessary to determine whether there are any missing reference features in the feature matrix. The following two methods are proposed to obtain the fused features by fusing the adjacency matrix and the feature matrix, for the two cases of missing reference features and no missing reference features.

[0171] Method 1, for cases where there are no missing reference features:

[0172] In response to the absence of missing reference features in the feature matrix, and after obtaining a preset number of updates, the feature matrix and the adjacency matrix are aggregated using a graph neural network to obtain a target feature matrix composed of target features after the feature matrix is ​​updated. Then, multiple target features corresponding to multiple features in the video to be processed are extracted from the target feature matrix, and the multiple target features are fused to obtain the fused features of the video to be processed.

[0173] Specifically, the process of aggregating the feature matrix and the adjacency matrix by a preset number of updates can be seen in the following formula.

[0174]

[0175] in, Let A be the adjacency matrix and I be the identity matrix. for The degree matrix, where l is the layer number or update count, W l H represents the learnable parameters of the l-th layer. l is the feature matrix input to the l-th layer.

[0176] By setting a preset number of updates, i.e., the update of the l-th layer mentioned above, the updated target feature matrix is ​​obtained. Then, multiple target features belonging to the video to be processed in the obtained target feature matrix are concatenated, and the concatenated feature is used as the fusion feature of the video to be processed.

[0177] It is worth noting that the above splicing process can be represented as a process of splicing target features based on feature dimensions.

[0178] Method 2, for cases where missing reference features exist:

[0179] In response to the presence of missing reference features in the feature matrix, the connection coefficients related to the missing reference features in the adjacency matrix are adjusted to a specified value. Then, a preset number of updates is obtained. Through a graph neural network, the feature matrix and the adjusted adjacency matrix are aggregated for the preset number of updates to obtain the target feature matrix after the feature matrix is ​​updated. Then, the target feature matrix is ​​updated again according to the preset mask matrix and the preset scaling matrix to obtain the fused features of the video to be processed.

[0180] Specifically, the row and column values ​​of the adjacency matrix corresponding to the missing reference feature are adjusted to specified values. These specified values ​​can typically be 0, but can also be other values ​​determined based on the specific application. By adjusting them to these specified values, the aim is to prevent the information from the missing reference feature from affecting the generation of the final fused feature.

[0181] Then, a graph neural network is used, that is, the feature matrix and the adjacency matrix are aggregated by a preset number of updates using the calculation method of Formula 3 to obtain the target feature matrix after the feature matrix is ​​updated. Then, multiple target features belonging to the same video in the obtained target feature matrix are concatenated to obtain the concatenated target feature matrix. Then, according to the preset mask matrix and the preset scaling matrix, the concatenated target feature matrix is ​​updated again to obtain the fused features of the video to be processed.

[0182] Here, the process of updating the spliced ​​target feature matrix based on the preset mask matrix and the preset scaling matrix can be seen in Formula 4 below.

[0183]

[0184] Where M is a preset mask matrix, S is a preset scaling matrix, and H is the spliced ​​target feature matrix.

[0185] In addition, the above The above The above N is the total number of nodes, which is also the total number of features. This represents the dimension of the stitched target features belonging to the same video in the stitched target feature matrix.

[0186] Here, the mask matrix is ​​generally a matrix composed of 0s and 1s, with 0s generally corresponding to the missing parts, and the scaling matrix is ​​generally used to strengthen or weaken the features of the non-missing modalities.

[0187] Furthermore, this application also proposes a method for calculating a scaling matrix. Specifically, the calculation formula for each row element in the scaling matrix can be found in Formula 5 below.

[0188]

[0189] Among them, S i p represents the element in the i-th row of the scaling matrix, where p is the proportion of elements in the i-th row that are set to the specified value, which is generally 0.

[0190] Using the above method, the target feature matrix of the splicing is updated again by using a preset mask matrix and a preset scaling matrix to obtain the updated target feature matrix. The splicing target features corresponding to the video to be processed are extracted from the updated target feature matrix and used as the fusion features of the video to be processed.

[0191] Step 104: Using the fusion features, determine the recognition result of the target object.

[0192] In this embodiment, a preset model can be used to identify fused features, obtain identification results, and use these results as the identification results of target objects in the video to be processed. Here, the identification results may include the identity information or attribute information of the target object.

[0193] Specifically, a linear classification layer can be used to classify the fused features to obtain the classification result. The obtained classification result is then used as the final recognition result for the target object in the video to be processed, thus completing the target recognition based on multimodal feature fusion.

[0194] The above method can be used to fuse features of different modalities of a target object and obtain fused features, which solves the problem of low target recognition accuracy caused by single modal features in existing technologies. Furthermore, by combining reference features to fuse the extracted features of different modalities, the accuracy of target recognition can be effectively improved.

[0195] Based on the technical solutions provided in the embodiments of this application, the following technical effects can be achieved:

[0196] 1. In the process of fusing multiple features of the video to be processed, the fusion of reference features makes the fused features of the final video to be processed more discriminative and robust. Based on the fused features, target recognition is performed, which effectively improves the accuracy and recognition rate of the final recognition and further reduces the false alarm rate.

[0197] 2. After determining the feature matrix composed of multiple features of the video to be processed and reference features, the adjacency matrix corresponding to the feature matrix is ​​determined. A method for aggregating the feature matrix and the adjacency matrix is ​​proposed, which can effectively improve the quality of the generated fused features and the target recognition effect.

[0198] 3. A method for handling missing reference features is proposed. By modifying the corresponding adjacency matrix, the information propagation of the reference features of the missing modality is organized. The target feature matrix is ​​then updated according to the pre-designed mask matrix and scaling matrix, which effectively solves the problem of missing modal features and can be applied to a wider range of scenarios.

[0199] Based on the same inventive concept, this application also provides a target recognition apparatus for fusing extracted modal features with reference features to obtain fused features of the video to be processed, and for performing target recognition based on the fused features, effectively improving the accuracy of target recognition. This addresses the problem of low target recognition accuracy in existing technologies due to single modal features. See [link to related document]. Figure 6 The device includes:

[0200] Extraction module 601 extracts multiple features of different modalities of the target object in the video to be processed;

[0201] The determining module 602 determines the reference features corresponding to the video to be processed; wherein, the reference features are determined based on the features of multiple reference videos, and the reference videos are features of videos having features of at least one of the different modalities;

[0202] The fusion module 603 fuses the multiple features of the video to be processed based on the reference features to obtain the fused features of the video to be processed.

[0203] The recognition module 604 uses the fusion features to determine the recognition result of the target object.

[0204] In one possible design, the extraction module 601 is specifically used for:

[0205] Extract a first set of images from the video to be processed, and perform the following operations on each image in the first set of images:

[0206] Calculate the similarity value between a single image and each image in the first image set. If all similarity values ​​are greater than a preset similarity threshold, then add the single image to the second image set.

[0207] In the second image set, features of different modalities of the target object are extracted from each image;

[0208] Weighted summation is performed on multiple features of the same modality extracted from each image to calculate one feature of the same modality, thereby obtaining multiple features of different modalities;

[0209] The calculated features of the different modalities are used as the features of the different modalities of the video to be processed.

[0210] In one possible design, the extraction module 601 is specifically used for:

[0211] Extract multiple images from the video to be processed as a third image set, and calculate the image quality score of each image in the third image set;

[0212] Extract all images with quality scores greater than a preset threshold to form a first image set.

[0213] In one possible design, the extraction module 601 is further configured to perform feature encoding on the video to be processed to obtain video features corresponding to the video to be processed;

[0214] The video features are added one by one to each of the multiple features to obtain the encoded features corresponding to each feature, and the obtained multiple encoded features are used as multiple features of the video to be processed.

[0215] In one possible design, the determining module 602 is specifically used for:

[0216] Calculate the similarity value between each of the multiple features and each feature in the preset video to obtain multiple similarity values ​​for each of the multiple features;

[0217] According to the magnitude of the similarity value, the multiple similarity values ​​of each feature among the multiple features are arranged, and the preset video corresponding to the similarity value arranged at the target position is taken as the reference video;

[0218] Features of different modalities are extracted from each reference video, and the extracted features are used as reference features for the video to be processed.

[0219] In one possible design, the determining module 602 is specifically used for:

[0220] Determine whether each reference video contains missing features of the different modalities;

[0221] If not, then extract the features of the different modalities in each of the reference videos, and use the extracted features as the reference features corresponding to the video to be processed;

[0222] If so, then extract the features of the different modalities in each of the reference videos, fill the extracted missing features with a specified vector, and use the filled features of the different modalities as the reference features corresponding to the video to be processed.

[0223] In one possible design, the fusion module 603 is specifically used for:

[0224] Determine a feature matrix composed of the plurality of features and the reference features;

[0225] Obtain the adjacency matrix corresponding to the feature matrix; wherein, the adjacency matrix represents the connection relationship between different features in the feature matrix for fusion;

[0226] By aggregating the feature matrix and the adjacency matrix, the fused features of the video to be processed are obtained.

[0227] In one possible design, the fusion module 603 is specifically used for:

[0228] Determine the connection coefficients between each feature in the plurality of features and each feature in the feature matrix for fusion;

[0229] Based on the determined connection coefficients, an adjacency matrix composed of the determined connection coefficients is obtained.

[0230] In one possible design, the fusion module 603 is specifically used for:

[0231] In response to the absence of missing reference features in the feature matrix, a preset number of updates is obtained;

[0232] By using a graph neural network, the feature matrix and the adjacency matrix are aggregated after the preset number of updates to obtain the target feature matrix after the feature matrix is ​​updated; wherein, the target feature matrix is ​​composed of target features;

[0233] In the target feature matrix, multiple target features corresponding to the multiple features are extracted, and the multiple target features are fused to obtain the fused features of the video to be processed.

[0234] In one possible design, the fusion module 603 is specifically used for:

[0235] In response to the presence of a missing reference feature in the feature matrix, the connection coefficients associated with the missing reference feature in the adjacency matrix are adjusted to a specified value;

[0236] By using a graph neural network, the feature matrix and the adjusted adjacency matrix are aggregated after a preset number of updates to obtain the target feature matrix after the feature matrix is ​​updated.

[0237] The target feature matrix is ​​updated again based on the preset mask matrix and the preset scaling matrix to obtain the fused features of the video to be processed.

[0238] Based on the above device and the above method, different modal features extracted by combining reference features are fused to obtain the fused features of the video to be processed, and target recognition is performed based on the fused features. This solves the problem that the target recognition accuracy is low due to single modal features in the existing technology. Based on this method, not only can the quality of the fused features be effectively improved, but the accuracy of target recognition based on the fused features can also be effectively improved.

[0239] Based on the same inventive concept, this application also provides an electronic device that can realize the function of the aforementioned target recognition device. (Refer to...) Figure 7 The electronic device includes:

[0240] At least one processor 701 and a memory 702 connected to at least one processor 701. In this embodiment, the specific connection medium between the processor 701 and the memory 702 is not limited. Figure 7 The example shown is the connection between processor 701 and memory 702 via bus 700. Bus 700 is... Figure 7The connections between other components are indicated by thick lines and are for illustrative purposes only, not as limiting information. The 700 bus can be divided into address bus, data bus, control bus, etc., for ease of representation. Figure 7 The term is represented by a single thick line, but this does not imply that there is only one bus or one type of bus. Alternatively, the processor 701 can also be called a controller; there is no restriction on the name.

[0241] In this embodiment, memory 702 stores instructions executable by at least one processor 701. By executing the instructions stored in memory 702, at least one processor 701 can perform the target recognition method described above. Processor 701 can implement... Figure 6 The functions of each module in the device shown.

[0242] The processor 701 is the control center of the device. It can connect to various parts of the control device through various interfaces and lines. By running or executing instructions stored in memory 702 and calling data stored in memory 702, the processor can perform various functions and process data, thereby monitoring the device as a whole.

[0243] In one possible design, processor 701 may include one or more processing units. Processor 701 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into processor 701. In some embodiments, processor 701 and memory 702 may be implemented on the same chip; in some embodiments, they may also be implemented on separate chips.

[0244] The processor 701 can be a general-purpose processor, such as a central processing unit (CPU), digital signal processor, application-specific integrated circuit, field-programmable gate array or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the target recognition method disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0245] Memory 702, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory 702 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. Memory 702 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. In the embodiments of this application, memory 702 can also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0246] By designing and programming the processor 701, the code corresponding to the target recognition method described in the foregoing embodiments can be embedded into the chip, enabling the chip to execute the code during operation. Figure 1 The steps of the target recognition method in the illustrated embodiment are described below. How to design and program the processor 701 is a technique well-known to those skilled in the art and will not be elaborated upon here.

[0247] Based on the same inventive concept, embodiments of this application also provide a storage medium storing computer instructions that, when executed on a computer, cause the computer to perform the target recognition method described above.

[0248] In some possible implementations, various aspects of the target identification method provided in this application may also be implemented in the form of a program product, which includes program code that, when the program product is run on a device, causes the control device to perform the steps in the target identification method according to the various exemplary embodiments of this application described above.

[0249] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0250] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0251] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0252] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0253] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for target recognition, characterized in that, The method includes: Extract multiple features of different modalities of the target object in the video to be processed; Calculate the similarity value between each feature in the plurality of features and each feature in the preset video to obtain multiple similarity values ​​for each feature in the plurality of features; arrange the multiple similarity values ​​of each feature in the plurality of features according to the magnitude of the similarity values, and take the preset video corresponding to the similarity value at the target position as the reference video; extract the features of the different modalities in each reference video, and use the extracted features as the reference features corresponding to the video to be processed; wherein, the reference video is a video having features of at least one of the different modalities; Based on the reference features, the multiple features of the video to be processed are fused to obtain the fused features of the video to be processed; The identification result of the target object is determined using the fusion features.

2. The method as described in claim 1, characterized in that, The extraction of multiple features of different modalities of the target object in the video to be processed includes: Extract a first set of images from the video to be processed, and perform the following operations on each image in the first set of images: Calculate the similarity value between a single image and each image in the first image set. If all similarity values ​​are greater than a preset similarity threshold, then add the single image to the second image set. In the second image set, features of different modalities of the target object are extracted from each image; Weighted summation is performed on multiple features of the same modality extracted from each image to calculate one feature of the same modality, thereby obtaining multiple features of different modalities; The calculated features of the different modalities are used as the features of the different modalities of the video to be processed.

3. The method as described in claim 2, characterized in that, The step of extracting the first image set from the video to be processed includes: Extract multiple images from the video to be processed as a third image set, and calculate the image quality score of each image in the third image set; Extract all images with quality scores greater than a preset threshold to form a first image set.

4. The method as described in claim 1, characterized in that, After extracting multiple features of different modalities of the target object in the video to be processed, the process further includes: The video to be processed is feature-encoded to obtain the video features corresponding to the video to be processed. The video features are added one by one to each of the multiple features to obtain the encoded features corresponding to each feature, and the obtained multiple encoded features are used as multiple features of the video to be processed.

5. The method as described in claim 1, characterized in that, The step of extracting features of different modalities from each reference video and using the extracted features as reference features for the video to be processed includes: Determine whether each reference video contains missing features of the different modalities; If not, then extract the features of the different modalities in each of the reference videos, and use the extracted features as the reference features corresponding to the video to be processed; If so, then extract the features of the different modalities in each of the reference videos, fill the extracted missing features with a specified vector, and use the filled features of the different modalities as the reference features corresponding to the video to be processed.

6. The method according to any one of claims 1-5, characterized in that, The process of fusing multiple features of the video to be processed based on the reference features to obtain the fused features of the video to be processed includes: Determine a feature matrix composed of the plurality of features and the reference features; Obtain the adjacency matrix corresponding to the feature matrix; wherein, the adjacency matrix represents the connection relationship between different features in the feature matrix for fusion; By aggregating the feature matrix and the adjacency matrix, the fused features of the video to be processed are obtained.

7. The method as described in claim 6, characterized in that, The step of obtaining the adjacency matrix corresponding to the feature matrix includes: Determine the connection coefficients between each feature in the plurality of features and each feature in the feature matrix for fusion; Based on the determined connection coefficients, an adjacency matrix composed of the determined connection coefficients is obtained.

8. The method as described in claim 6, characterized in that, The process of aggregating the feature matrix and the adjacency matrix to obtain the fused features of the video to be processed includes: In response to the absence of missing reference features in the feature matrix, a preset number of updates is obtained; By using a graph neural network, the feature matrix and the adjacency matrix are aggregated after the preset number of updates to obtain the target feature matrix after the feature matrix is ​​updated; wherein, the target feature matrix is ​​composed of target features; In the target feature matrix, multiple target features corresponding to the multiple features are extracted, and the multiple target features are fused to obtain the fused features of the video to be processed.

9. The method as described in claim 6, characterized in that, The process of aggregating the feature matrix and the adjacency matrix to obtain the fused features of the video to be processed includes: In response to the presence of a missing reference feature in the feature matrix, the connection coefficients associated with the missing reference feature in the adjacency matrix are adjusted to a specified value; By using a graph neural network, the feature matrix and the adjusted adjacency matrix are aggregated after a preset number of updates to obtain the target feature matrix after the feature matrix is ​​updated. The target feature matrix is ​​updated again based on the preset mask matrix and the preset scaling matrix to obtain the fused features of the video to be processed.

10. A target recognition device, characterized in that, The device includes: The extraction module extracts multiple features of different modalities of the target object in the video to be processed; The determination module calculates the similarity value between each feature in the plurality of features and each feature in the preset video, obtaining multiple similarity values ​​for each feature; it then arranges the multiple similarity values ​​for each feature according to their magnitude, and takes the preset video corresponding to the similarity value at the target position as a reference video; it extracts the features of different modalities from each reference video, and uses the extracted features as reference features for the video to be processed; wherein, the reference video is a video with features of at least one of the different modalities; The fusion module, based on the reference features, fuses the multiple features of the video to be processed to obtain the fused features of the video to be processed; The identification module uses the fused features to determine the identification result of the target object.

11. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, when executing a computer program stored in the memory, implements the method steps of any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method steps of any one of claims 1-9.

Images