Target identification system based on multi-modal information

Abstract

The embodiment of the invention discloses a target recognition system based on multi-modal information, and the system comprises the steps: extracting object feature information from to-be-recognized information, and determining target recognition object information based on the object feature information.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computers, and specifically to a target recognition system based on multimodal information. Background Art

[0002] Target recognition refers to identifying the target object from the information to be recognized, for example, identifying the speaker from the video, such as determining whether the person in the picture is speaking, and if there is a speaker, then find its position in the picture simultaneously. As one of the sub-directions in the computer field, this task has gradually attracted the attention of researchers in recent years due to its huge technical challenges and broad market application prospects. Summary of the invention

[0003] An embodiment of the present application provides a target recognition system based on multimodal information.

[0004] In a first aspect, an embodiment of the present application provides a method, including:

[0005] Obtaining information to be identified;

[0006] Extracting features from the information to be identified to obtain object feature information;

[0007] Based on the object feature information, target recognition object information is determined.

[0008] In a second aspect, an embodiment of the present application further provides a system, including:

[0009] An acquisition module, used for acquiring information to be identified;

[0010] An extraction module, used for extracting features from the information to be identified to obtain object feature information;

[0011] The determination module is used to determine the target recognition object information based on the object feature information.

[0012] In a third aspect, an embodiment of the present application further provides a device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps in the above method when executing the computer program.

[0013] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps in the above method are implemented.

[0014] In a fifth aspect, the embodiments of the present application further provide a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in various optional implementations described in the embodiments of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0016] Figure 1 is a schematic diagram of a scenario of a target recognition method provided in an embodiment of the present application;

[0017] Figure 2 is a flow chart of a target recognition method provided in an embodiment of the present application;

[0018] Figure 3 is a first structural diagram of an object recognition model provided in an embodiment of the present application;

[0019] Figure 4 is a flow chart of identifying target and object information based on object feature information provided by an embodiment of the present application;

[0020] Figure 5 is a second structural diagram of the object recognition model provided in an embodiment of the present application;

[0021] Figure 6 is a third structural diagram of the object recognition model provided in an embodiment of the present application;

[0022] Figure 7 is a schematic diagram of a differential image obtained by differential operation of adjacent sequence frames provided in an embodiment of the present application;

[0023] Figure 8 This is a recognition effect diagram of using a differential image as the input of a first encoding module of an object recognition model provided in an embodiment of the present application;

[0024] Fig. 9 is a schematic diagram of mask marking provided in an embodiment of the present application;

[0025] Fig.10 It is a schematic diagram of recognition effect under different masking rates provided by the embodiment of the present application;

[0026] Fig.11 It is a schematic diagram of constructing positive samples and negative samples provided in the embodiments of the present application;

[0027] Fig.12 is a schematic diagram of the structure of the target recognition system provided in the embodiment of the present application;

[0028] Fig.13 It is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in this application to clearly and completely describe the technical solutions in this application. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments in the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0030] The embodiments of the present application provide a method, system, computer device and computer-readable storage medium for target recognition based on multimodal information. Specifically, the computer device includes a terminal device or a server, wherein the terminal device includes but is not limited to a desktop computer, a laptop computer, a mobile phone, a tablet computer and other devices, and the server can be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDN, Content Delivery Network), and big data and artificial intelligence platforms, etc. The server can be directly or indirectly connected via wired or wireless communication.

[0031] See also Figure 1 Taking the target identification method executed by the terminal device as an example, the server executing the target identification method can be understood by referring to the steps of executing the target identification method by the terminal device, wherein the specific execution process of the target identification method executed by the terminal device is as follows:

[0032] The terminal device 10 obtains the information to be identified, performs feature extraction on the information to be identified, obtains object feature information, and determines target identification object information based on the object feature information.

[0033] It can be understood that by extracting the features of the information to be identified to obtain object feature information and determining the target identification object information based on the object feature information, the accuracy of target object identification is improved.

[0034] It should be noted that the order of description of the following embodiments is not intended to limit the priority order of the embodiments.

[0035] See also Figure 2 , Figure 2 The following is a flow chart of a target recognition method provided in an embodiment of the present application. The specific flow of the target recognition method can be as follows:

[0036] 101. Obtain information to be identified.

[0037] It is understandable that the information to be identified is information from which a target identification object needs to be identified, and the target identification object includes people, animals, vehicles, icons, numbers, texts, etc. Based on different target identification requirements, the target identification object can be different. For example, for person identification, the target identification object to be identified is a person, and for vehicle identification, the target identification object to be identified is a vehicle.

[0038] Accordingly, the information to be identified may be text information, image information, audio information or video information, for example, identifying a target text from an original text, or identifying a target image or target face from an image, or identifying a face from video information, etc.

[0039] Among them, the embodiment of the present application takes the information to be identified as video information (referring to media data including pictures and audio) and the target identification object as a person as an example. Identifying the target identification object from the information to be identified is understood as identifying the target person from the video information, for example, identifying a speaker who is speaking from the video information.

[0040] It can be understood that if the information to be identified is video information, then the information to be identified can be video information containing a conversation scene, wherein the conversation scene refers to a scene where speaking occurs, such as a meeting scene, a party scene, or an academic exchange scene, etc. The embodiment of the present application can be applied to a multi-user speaking scene to filter out the target person who is currently speaking from multiple users.

[0041] 102. Perform feature extraction on the information to be identified to obtain object feature information.

[0042] It should be noted that the object characteristic information is characteristic information reflecting the target identification object in the information to be identified, and the object characteristic information is used to distinguish various objects in the information to be identified so as to identify the target identification object therefrom.

[0043] It is understandable that, compared with unimodal information, the accuracy of multimodal information in recognition is improved. Among them, unimodal information is data of only one type, such as one of text, image, audio, video, electromagnetic signal and other data information. Multimodal information is data information including at least two unimodal information. Furthermore, multimodal information is suitable for complex tasks that require the integration of multiple information sources, such as sentiment analysis, robot interaction, autonomous driving, etc. By integrating information of multiple modalities, higher performance and accuracy can usually be achieved in tasks. For example, on the basis of image modality, combining audio modality can improve the accuracy of target recognition object recognition. Therefore, in an embodiment of the present application, the object feature information includes at least face information of image modality and audio information of audio modality, etc. Specifically, face information includes face sequence information, which refers to sequence information composed of face images; audio information includes audio feature information, which includes audio timbre, pitch, frequency and intensity and other feature information.

[0044] 103. Determine target recognition object information based on the object feature information.

[0045] Among them, since the object feature information is used to represent the features of the object, it is suitable to use the object feature information to distinguish it from other objects. Therefore, the object feature information can be used to identify the target identification object information from the information to be identified.

[0046] In summary, by extracting the features of the information to be identified, object feature information is obtained, and target identification object information is determined based on the object feature information, thereby improving the accuracy of target identification object identification. After the target identification object is identified, the target identification object can be marked by a marking box or the like. For example, the target identification object can be marked by a marking box in the video screen, or the target identification object can be focused by focusing the lens or window to locate the speaker.

[0047] Optionally, in order to improve the accuracy of target object recognition, a deep learning model can be used to identify the target object in the information to be recognized. For example, see Figure 3 As shown, Figure 3It is a first structural schematic diagram of an object recognition model provided in an embodiment of the present application, wherein the object recognition model includes a first encoding module, a second encoding module, a fusion module and a timing processing module, wherein the input end of the first encoding module is configured to receive the first model input information of the object recognition model, the input end of the second encoding module is configured to receive the second model input information of the object recognition model, the output end of the first encoding module is connected to the input end of the fusion module, the output end of the second encoding module is connected to the input end of the fusion module, the output end of the fusion module is connected to the input end of the timing processing module, and the output end of the timing processing module is configured to output the model output information of the object recognition model.

[0048] Among them, the first model input information of the object recognition model is the image information in the information to be recognized, and the second model input information of the object recognition model is the audio information in the information to be recognized; the first encoding module is configured to perform image encoding on the image information to obtain an image embedding representation; the second encoding module is configured to perform audio encoding on the audio information to obtain an audio embedding representation; the fusion module is configured to fuse the image embedding representation and the audio embedding representation to obtain a fused feature representation; the timing processing module is configured to establish a timing representation for the fused feature representation in the time dimension.

[0049] Among them, in an embodiment of the present application, the first encoding module and the second encoding module can be constructed using benchmark networks such as ResNet101, Transformer, GNN, etc., the timing processing module can be modeled using operators such as 3D convolution, RNN / LSTM, and the fusion module can use benchmark networks such as ResNet, DANet, etc.

[0050] It can be understood that, through the object recognition model, the target recognition object information of the information to be recognized can be recognized by utilizing the multimodal information (including at least image information and audio information) in the information to be recognized.

[0051] For example, the embodiment of the present application takes the identification of a speaker from video information (information to be identified) as an example. Then, the target identification object information is identified from the information to be identified through the object recognition model, which can be understood as identifying the speaker from the video information by using the image information and audio information corresponding to the video information through the object recognition model.

[0052] Among them, since the acquisition of mouth information is more dependent on the visual picture, and whether the face is deflected, that is, whether the mouth shape information can be collected, has a greater impact on speaker recognition, therefore, in the embodiment of the present application, the object posture information can be introduced to analyze whether there is a side face situation to improve the accuracy of target recognition, that is, optionally, in some embodiments of the present application, the step of "determining the target recognition object information based on the object feature information" includes:

[0053] Determining object posture information based on the object feature information;

[0054] If the object posture information meets the first preset condition, the object feature information is identified and processed using the object recognition model to obtain target recognition object information; and / or,

[0055] If the object posture information does not meet the first preset condition, feature analysis and recognition processing is performed on the object feature information to obtain the target recognition object information.

[0056] Among them, the object posture information refers to the facial posture information of a person, that is, the facial deflection angle information. The object posture information can be extracted from the facial information corresponding to the information to be identified through computer vision tasks, for example, the facial deflection angle is calculated through face detection, feature extraction, posture estimation, deflection angle calculation and other steps.

[0057] Among them, when the face deflection angle is small, that is, the mouth movement information can be obtained from the picture, the target recognition object information obtained based on the object feature information recognition has a higher accuracy; conversely, when the face deflection angle is large, that is, the mouth movement information cannot be obtained from the picture, it is difficult to ensure the accuracy of the target recognition object information obtained based on the object feature information recognition.

[0058] Therefore, in order to improve the accuracy of the target object information finally identified, whether the face deflection angle exceeds the set angle is used to determine whether the target object information obtained by the object recognition model recognition process is directly used as the final recognition result. That is, in the embodiment of the present application, the first preset condition includes that the face deflection angle is less than the set angle, and "the object posture information satisfies the first preset condition" indicates that the face deflection angle determined by the information to be identified is less than the set angle, and conversely, "the object posture information does not meet the first preset condition" indicates that the face deflection angle determined by the information to be identified is greater than or equal to the set angle. Among them, the set angle is pre-set, for example, the set angle is set according to the recognition accuracy under different angles tested previously.

[0059] Furthermore, when the face deflection angle does not meet the first preset condition, it is necessary to further perform feature analysis and recognition processing on the object feature information to improve the accuracy of the target recognition object information. For example, in an embodiment of the present application, since the audio information of different people is different, for example, the timbre of different people is different, therefore, when the face deflection angle does not meet the first preset condition, the accuracy of the target recognition object information can be improved by analyzing the collected object timbre information, that is, optionally, in some embodiments of the present application, the step of "if the object posture information does not meet the first preset condition, perform feature analysis and recognition processing on the object feature information to obtain the target recognition object information" includes:

[0060] If the object posture information does not satisfy the first preset condition, determining the object timbre information based on the object feature information; and / or,

[0061] If the object timbre information and the historical timbre information of the previously identified historical object information meet the second preset condition, the object feature information is identified and processed using the object recognition model to obtain the target identification object information; and / or,

[0062] If the object timbre information and the historical timbre information of the previously recognized historical object information do not meet the second preset condition, the target recognition object information is determined according to the historical object information.

[0063] Among them, the object timbre information is also the timbre information of the object to be identified, and the object to be identified is the object contained in the information to be identified. It can be understood that the object to be identified is an object that needs to be determined as a target identification object through object posture information and object timbre information in the embodiment of the present application. Among them, timbre is also called sound quality or sound quality, which refers to the unique quality and characteristics of sounds emitted by different sound sources. It is determined by the harmonic components of the sound and their relative intensities. These harmonic components are composed of waves of multiple frequencies generated when the sound source vibrates. In physics, timbre is determined by the spectral characteristics of sound waves, including fundamental frequency and harmonic frequency. There are differences in the timbre of different speakers. In real environments, people usually use timbre to distinguish who is speaking.

[0064] Among them, the historical object information is the information of the target recognition object identified at the previous moment. The target recognition object identified at the previous moment is the target recognition object identified for the video information at the previous moment. It can be understood that in order to improve the accuracy of target recognition object recognition, the historical object usually uses the historical object corresponding to the historical timestamp that is closer to the current timestamp. For example, for the target recognition task at the current moment (timestamp T), the historical object is the recognition result at the previous moment such as timestamp T-1 or T-2. It can be understood that the recognition result at the previous moment is also obtained through comprehensive judgment and processing of operations such as face deflection angle, timbre analysis, and combined with recognition of the object recognition model. That is, the processing flow of the target recognition task at the current moment is consistent with the processing flow of the historical object information obtained by the recognition processing at the previous moment.

[0065] Correspondingly, the historical timbre information is the timbre information of the target recognition object recognized at the previous moment, which is the sound quality or timbre of the target recognition object recognized at the previous moment.

[0066] Among them, if the object timbre information is consistent with the historical timbre information of the previously recognized historical object information, it is highly likely that the current speaker is still the historical object recognized before, so the historical object information can be directly used as the target recognition object information, and vice versa, if the object timbre information is inconsistent with the historical timbre information of the previously recognized historical object information, it is highly likely that the current speaker is different from the previously recognized historical object. Therefore, the embodiment of the present application uses timbre information to analyze the speaker when the face deflection angle is large, which can improve the accuracy of the target recognition object information.

[0067] Correspondingly, in the embodiment of the present application, the second preset condition refers to the inconsistency of timbre information, then "the object timbre information and the historical timbre information of the previously identified historical object information meet the second preset condition" can be understood as the object timbre information being inconsistent with the historical timbre information, and "the object timbre information and the historical timbre information of the previously identified historical object information do not meet the second preset condition" can be understood as the object timbre information being consistent with the historical timbre information.

[0068] For example, see Figure 4 , Figure 4 This is a flow chart of identifying target object information based on object feature information provided by an embodiment of the present application. The process of identifying and obtaining historical object information at a previous moment can be understood by reference, specifically:

[0069] 201. Determine whether the face deflection angle is large. If so, execute step 202. If not, execute step 204.

[0070] Among them, the face deflection angle corresponds to the object posture information, which is obtained by extracting the face information.

[0071] 202, determining whether the timbre information is consistent with the timbre information of the previously identified historical object, if consistent, executing step 203, if not, executing step 204;

[0072] The timbre information is obtained by extracting features of the audio information. For example, the timbre features or timbre information corresponding to the audio information are identified through steps such as preprocessing of the audio information, feature extraction, and timbre recognition.

[0073] 203. Determine the historical object as a target identification object;

[0074] 204. Identify a target recognition object in a conversation scene corresponding to the video information through an object recognition model.

[0075] The target recognition object is the object corresponding to the target recognition object information, such as a speaker.

[0076] That is, when the face deflection angle is large and the timbre is consistent with the timbre of the previously recognized historical object, the historical object is determined to be the current speaker, and the historical object is determined to be the target recognition object; conversely, if the face deflection angle is large and the timbre is inconsistent with the timbre of the previously recognized historical object, it is considered that the historical object is not the current speaker, and the historical object cannot be determined as the target recognition object. Accordingly, the reference object with a higher confidence level identified by the object recognition model can be used as the target recognition object to maximize the accuracy of the recognition result through confidence.

[0077] Among them, in the embodiment of the present application, after the target recognition object is identified, the target recognition object is displayed in the form of an observation window, for example, an observation window is added to the screen to lock the target recognition object. Therefore, when judging whether the timbre is consistent with the timbre of the previously recognized historical object, it can be determined by comparing the timbre corresponding to the continuous multiple observation windows corresponding to the previous historical object, so as to improve the accuracy of the judgment result of whether the timbre is consistent.

[0078] Among them, since the object recognition model based on multimodal information (the embodiment of the present application includes at least image information and audio information) often uses a dual branch to extract audio and video features, for example, by designing a first encoding module for images and a second encoding module for audio to extract audio and video features, the computing power (i.e., GFLOPS) is improved. Therefore, in the embodiment of the present application, a basic operator can be used to construct an object recognition model, please refer to Figure 5 , Figure 5It is a second structural schematic diagram of the object recognition model provided by an embodiment of the present application, wherein the first encoding module includes a first combination unit, a first pooling unit, a first residual unit, a second combination unit and a second pooling unit in sequence, the input end of the first combination unit is configured to receive the first model input information of the object recognition model, the output end of the first combination unit is connected to the input end of the first pooling unit, the output end of the first pooling unit is connected to the input end of the first residual unit, the output end of the first residual unit is connected to the input end of the second combination unit, the output end of the second combination unit is connected to the input end of the second pooling unit, and the output end of the second pooling unit is connected to the input end of the fusion module;

[0079] The second encoding module includes a third combination unit, a second residual unit, a fourth combination unit, and a conversion unit, wherein the input end of the third combination unit is configured to receive the second model input information of the object recognition model, the output end of the third combination unit is connected to the input end of the second residual unit, the output end of the second residual unit is connected to the input end of the fourth combination unit, the output end of the fourth combination unit is connected to the input end of the conversion unit, and the output end of the conversion unit is connected to the input end of the fusion module;

[0080] The fusion module comprises a feature connection unit, the input end of the feature connection unit is respectively connected to the output end of the second pooling unit and the output end of the conversion unit, and the output end of the feature connection unit is connected to the input end of the timing processing module;

[0081] The timing processing module sequentially comprises a separation convolution unit, a fifth combination unit and a full connection unit, the input end of the separation convolution unit is connected to the output end of the feature connection unit, the output end of the separation convolution unit is connected to the input end of the fifth combination unit, the output end of the fifth combination unit is connected to the input end of the full connection unit, and the output end of the full connection unit is configured to output the model output information of the object recognition model;

[0082] Among them, the first combination unit, the third combination unit, the fourth combination unit and the fifth combination unit are combination units with the same structure and different parameters, and the first residual unit and the second residual unit are residual units with the same structure and different parameters.

[0083] For details, please refer to Figure 6 , Figure 6: is a third structural schematic diagram of an object recognition model provided by an embodiment of the present application, wherein the combination units corresponding to the first combination unit, the third combination unit, the fourth combination unit and the fifth combination unit include a first convolutional layer (including a two-dimensional convolutional layer conv2D), a normalization layer (such as batch normalization BN) and an activation layer (such as ReLu, Sigmoid), the second combination unit includes a first convolutional layer and a normalization layer, the residual unit corresponding to the first residual unit and the second residual unit includes at least one residual subunit (InvRes) with the same structure and different parameters, the residual subunit includes a sixth combination unit, a seventh combination unit, an eighth combination unit and a feature addition layer, the separation convolution unit includes at least one separation convolution subunit (DSConv1D) with the same structure and the same parameters, and the separation convolution subunit includes a second convolutional layer (including a one-dimensional convolutional layer onv1D), a normalization layer and an activation layer;

[0084] The sixth and seventh combination units are combination units with the same structure and different parameters as the first combination unit, and the eighth combination unit is a combination unit with the same structure and different parameters as the second component unit. The first pooling unit includes a maximum pooling layer (MaxPool), the second pooling unit includes an average pooling layer (AvgPool), the conversion unit includes a matrix conversion layer (trans), and the feature connection unit includes a feature concatenation layer (concat).

[0085] Among them, the same structure means that the layer structures that constitute each unit and sub-unit are the same. For example, the first combination unit and the third combination unit are both obtained by combining the first convolutional layer, the normalization layer and the activation layer. Different parameters mean that the dimensions of each unit or sub-unit are different. For example, the parameters of the first combination unit are T×32×56×56, and the parameters of the third combination unit are 16×7×(4T).

[0086] As can be seen from the above, the object recognition model is mainly composed of basic operators such as convolutional layers, normalization layers, activation layers, and fully connected layers, which simplifies the model structure and the complexity of model processing, and reduces the processing power. For example, under the premise of maintaining similar prediction effects, the entire network contains only 0.55M parameters, and a single inference requires only 0.40GFLOPS, which is far lower than the existing open source network structure parameters and computing scale (15M parameters, single inference computing of about 2.5GFLOPS). In the simulation test on the MTK-9653SoC, a single inference takes only 6.5ms.

[0087] Optionally, taking the face image as an example, since the face image input to the first encoding module of the object recognition model contains more noise, for example, facial attribute information such as facial skin color, expression, and facial feature proportions, the features extracted by the first encoding module contain more interference information, which makes it easy for learning and analysis to have deviations. Therefore, in an embodiment of the present application, the differential image of adjacent sequence frames is used instead of the face image as the input of the first encoding module, so that the feature information extracted by the first encoding module pays more attention to the face movement. That is, optionally, in some embodiments of the present application, the image information is the differential image formed by adjacent sequence frames corresponding to the information to be identified.

[0088] The difference image is calculated by the difference operation of adjacent sequence frames, such as Figure 7 As shown, Figure 7 It is a schematic diagram of a differential image obtained by differential operation of adjacent sequence frames provided in an embodiment of the present application. Since the differential operation well preserves the motion trend information of the face area, the first encoding module (such as the visual encoder) can more easily capture these motion features, and the differential operation also erases or weakens the performance features of the face area that are not related to motion, thereby significantly reducing the interference of static face representation and background information on the visual encoder and learning bias, so that the model converges faster and better.

[0089] For example, see Figure 8 As shown, Figure 8 This is a recognition effect diagram of the first encoding module of the object recognition model using the differential image as the input of the object recognition model provided in the embodiment of the present application. After using the differential image as the input of the first encoding module, compared with the method of directly using the face image, the indicators of each evaluation dimension are significantly higher than the original RGB input sequence.

[0090] Among them, the data annotation paradigm of the common target recognition (i.e., speaker detection) task is often to annotate continuously frame by frame and face by face, which will undoubtedly greatly increase the workload required for the data annotation link. Based on this problem, in the embodiments of the present application, the method of extracting sample multimodal information according to timestamps is adopted to reduce the amount of annotated data, thereby reducing the workload required for the data annotation link. That is, optionally, in some embodiments of the present application, the object recognition model is obtained by executing the following steps:

[0091] Obtain multimodal information of samples;

[0092] Extracting sample multimodal information with a portion of time stamps from the sample multimodal information according to the time stamps to obtain extracted multimodal information;

[0093] An object recognition model is generated by training based on the extracted multimodal information and sample annotation information corresponding to the extracted multimodal information.

[0094] For example, see Fig. 9 As shown, Fig. 9 is a schematic diagram of mask annotation provided by an embodiment of the present application. Weak annotation is used when annotating data, that is, only a portion of frames at the beginning, middle, and end of the speech segment need to be annotated, the timestamps of the annotated frames need to be recorded, and the corresponding ±1 masks are generated for such weakly annotated data (that is, according to the timestamp, 1 is taken at the annotated position of the sequence (such as Fig. 9 The dark part of the mask mark), and the rest of the positions are -1 (such as Fig. 9 The light-colored part of the mask annotation)) During model training, only the prediction results of the positions with a value of 1 in the sequence are used for supervision, and gradient feedback is performed to optimize the model. Compared with the traditional annotation method, the mask annotation method only extracts the sample multimodal information of some timestamps from the sequence table according to the timestamp from the sample multimodal information to obtain the extracted multimodal information, and only annotates these extracted multimodal information, which reduces the workload of annotation.

[0095] Optionally, in an embodiment of the present application, the mask rate (the proportion of frames with a value of -1 to all frames) and the position of the mask with a value of 1 can be set based on actual needs. For example, in order to improve accuracy, a portion of sequence frames at the beginning, middle, and end of the sample multimodal information can be selected and set as a mask with a value of 1.

[0096] See also Fig.10 As shown, Fig.10 It is a schematic diagram of recognition effects under different mask rates provided in an embodiment of the present application, wherein the recognition effect is displayed in the form of a row list, wherein the row parameters of the row list are face image, difference image and difference image with different mask rates (90%, ..., 10%), and the column parameters are face size facesize, accuracy ACC, area under the recall curve AUC, average precision AP, score F1, recall rate Recall and precision Precision.

[0097] Depend on Fig.10 It can be seen that the difference image has improved recognition effect compared with the face image, and even if the mask rate reaches 90% (that is, 90% of the sequence positions are not labeled), it still has a better recognition effect. Accordingly, in practical applications, the appropriate mask rate can be selected based on actual needs, such as the balance between accuracy and workload.

[0098] Among them, the current dialogue recognition model often uses the following positive and negative sample construction method during training: the positive sample is a speaker's audio and video sequence with synchronized audio and video, and the negative sample is a non-speaker's audio and video sequence. The negative samples generated by this construction method have very significant visual features (the face basically keeps its mouth closed). During the training process, the model can easily capture this type of data characteristics, resulting in overfitting, which limits the generalization ability of the model. In addition, in actual application scenarios, there are a large number of situations where people do not speak but the face screen shakes and the mouth opens and closes, resulting in inaccurate recognition results. Based on this, in an embodiment of the present application, negative samples are constructed based on asynchronous and heterogeneous methods of different modalities, thereby improving the recognition effect of the model.

[0099] That is, optionally, in some embodiments of the present application, the object recognition model is obtained by executing the following steps:

[0100] Acquire sample multimodal information, where the sample multimodal information includes first modality sample information and second modality sample information;

[0101] Constructing a positive sample according to the synchronized first modality sample information and second modality sample information;

[0102] Constructing a negative sample according to asynchronous first modality sample information and second modality sample information, or constructing a negative sample according to heterogeneous first modality sample information and second modality sample information;

[0103] The original recognition model is trained according to the constructed positive samples and negative samples to generate an object recognition model.

[0104] Among them, the object recognition model is a model obtained after the original recognition model is trained, that is, the object recognition model is obtained after the original recognition model is trained, wherein the model structure of the original recognition model and the object recognition model are the same, there are differences in the parameters in the model, and the parameters in the object recognition model are obtained through training.

[0105] Here, taking the face image sequence as the first modality sample information and the audio sequence as the second modality sample information as an example, Fig.11 As shown, Fig.11 It is a schematic diagram of constructing positive samples and negative samples provided in an embodiment of the present application, wherein the positive samples are configured using a synchronous strategy, and synchronization means that when a speaker speaks, his face image sequence (abbreviated as face sequence) matches the audio sequence in timestamp; the negative samples are configured using an asynchronous or heterogeneous strategy, wherein asynchronous means that when a speaker speaks, his face image sequence does not match the audio sequence in timestamp, such as offsetting the synchronous audio forward or backward so that the sound and picture are obviously out of sync, and heterogeneous means that the audio sequence and the face image sequence are not for the same speaker, such as extracting an equal-length audio sequence from other audio based on the duration of the face image sequence.

[0106] It can be understood that the embodiments of the present application construct negative samples by means of face image sequence + asynchronous audio or face image sequence + heterogeneous audio, which solves the problem of inaccurate recognition caused by obvious closed mouth state or incomplete negative samples when constructing current negative samples.

[0107] In summary, the embodiments of the present application extract recognition reference information from multimodal information, which helps to identify the target recognition object from the multimodal information based on the recognition reference information, simplifies the model structure and parameter quantity by simplifying the operators that constitute the object recognition model, and reduces the computational complexity of the object recognition model when it is applied.

[0108] Among them, the weak labeling and mask processing strategies are used to reduce the workload of the object recognition model during learning and training.

[0109] Among them, by using differential images instead of face images as the input of the visual branch (first encoding module) of the object recognition model, the features extracted by the first encoding module are reduced, and the feature interference during recognition is reduced. At the same time, the use of differential images in model training can also improve the model learning effect and the convergence quality of the model.

[0110] Among them, by adopting an asynchronous or heterogeneous strategy of the first modality sample information and the second modality sample information to construct negative samples, the problem that the negative samples in the prior art are mainly in an obvious closed-mouth state or the negative samples are incomplete is overcome, which helps to improve the accuracy of the recognition results of the object recognition model.

[0111] In order to better implement the target recognition method of the present application, the present application also provides a target recognition system based on the target recognition method. The meaning of the third target term is the same as that in the target recognition method, and the specific implementation details can refer to the description in the method embodiment.

[0112] See also Fig.12 , Fig.12 : is a schematic diagram of the structure of the target recognition system provided by the present application, wherein the target recognition system includes:

[0113] An acquisition module 301 is used to acquire information to be identified;

[0114] An extraction module 302 is used to extract features from the information to be identified to obtain object feature information;

[0115] A determination module 303 is used to determine target recognition object information based on the object feature information;

[0116] Preferably, the determination module 303 determines the target recognition object information based on the object feature information, including:

[0117] Determining object posture information based on the object feature information;

[0118] If the object posture information meets the first preset condition, the object feature information is identified and processed using the object recognition model to obtain target recognition object information; and / or,

[0119] If the object posture information does not meet the first preset condition, performing feature analysis and recognition processing on the object feature information to obtain the target recognition object information;

[0120] Preferably, the determination module 303 is used to perform feature analysis and recognition processing on the object feature information to obtain the target recognition object information if the object posture information does not meet the first preset condition, including:

[0121] If the object posture information does not satisfy the first preset condition, determining the object timbre information based on the object feature information; and / or,

[0122] If the object timbre information and the historical timbre information of the previously identified historical object information meet the second preset condition, the object feature information is identified and processed using the object recognition model to obtain the target identification object information; and / or,

[0123] If the object timbre information and the historical timbre information of the previously recognized historical object information do not meet the second preset condition, determining the target recognition object information according to the historical object information;

[0124] Preferably, the object recognition model comprises a first encoding module, a second encoding module, a fusion module and a timing processing module, the input end of the first encoding module is configured to receive the first model input information of the object recognition model, the input end of the second encoding module is configured to receive the second model input information of the object recognition model, the output end of the first encoding module is connected to the input end of the fusion module, the output end of the second encoding module is connected to the input end of the fusion module, the output end of the fusion module is connected to the input end of the timing processing module, and the output end of the timing processing module is configured to output the model output information of the object recognition model;

[0125] Preferably, the first model input information of the object recognition model is the image information in the information to be recognized, and the second model input information of the object recognition model is the audio information in the information to be recognized;

[0126] The first encoding module is configured to perform image encoding on the image information to obtain an image embedding representation;

[0127] The second encoding module is configured to perform audio encoding on the audio information to obtain an audio embedding representation;

[0128] The fusion module is configured to fuse the image embedding representation and the audio embedding representation to obtain a fused feature representation;

[0129] The time series processing module is configured to establish a time series representation for the fused feature representation in the time dimension;

[0130] Preferably, the first encoding module includes a first combination unit, a first pooling unit, a first residual unit, a second combination unit and a second pooling unit in sequence, the input end of the first combination unit is configured to receive the first model input information of the object recognition model, the output end of the first combination unit is connected to the input end of the first pooling unit, the output end of the first pooling unit is connected to the input end of the first residual unit, the output end of the first residual unit is connected to the input end of the second combination unit, the output end of the second combination unit is connected to the input end of the second pooling unit, and the output end of the second pooling unit is connected to the input end of the fusion module;

[0131] The second encoding module includes a third combination unit, a second residual unit, a fourth combination unit, and a conversion unit, wherein the input end of the third combination unit is configured to receive the second model input information of the object recognition model, the output end of the third combination unit is connected to the input end of the second residual unit, the output end of the second residual unit is connected to the input end of the fourth combination unit, the output end of the fourth combination unit is connected to the input end of the conversion unit, and the output end of the conversion unit is connected to the input end of the fusion module;

[0132] The fusion module comprises a feature connection unit, the input end of the feature connection unit is respectively connected to the output end of the second pooling unit and the output end of the conversion unit, and the output end of the feature connection unit is connected to the input end of the timing processing module;

[0133] The timing processing module sequentially comprises a separation convolution unit, a fifth combination unit and a full connection unit, the input end of the separation convolution unit is connected to the output end of the feature connection unit, the output end of the separation convolution unit is connected to the input end of the fifth combination unit, the output end of the fifth combination unit is connected to the input end of the full connection unit, and the output end of the full connection unit is configured to output the model output information of the object recognition model;

[0134] Among them, the first combination unit, the third combination unit, the fourth combination unit and the fifth combination unit are combination units with the same structure and different parameters, and the first residual unit and the second residual unit are residual units with the same structure and different parameters.

[0135] Preferably, the combination units corresponding to the first combination unit, the third combination unit, the fourth combination unit and the fifth combination unit include a first convolution layer, a normalization layer and an activation layer, the second combination unit includes a first convolution layer and a normalization layer, the residual units corresponding to the first residual unit and the second residual unit include at least one residual sub-unit with the same structure and different parameters, the residual sub-unit includes a sixth combination unit, a seventh combination unit, an eighth combination unit and a feature addition layer, the separation convolution unit includes at least one separation convolution sub-unit with the same structure and the same parameters, and the separation convolution sub-unit includes a second convolution layer, a normalization layer and an activation layer;

[0136] The sixth assembly unit and the seventh assembly unit are assembly units with the same structure and different parameters as the first assembly unit, and the eighth assembly unit is an assembly unit with the same structure and different parameters as the second assembly unit;

[0137] Preferably, the image information is a differential image formed by adjacent sequence frames corresponding to the information to be identified;

[0138] Preferably, the object recognition model is obtained by the determination module 303 by executing the following steps:

[0139] Acquire sample multimodal information, where the sample multimodal information includes first modality sample information and second modality sample information;

[0140] Constructing a positive sample according to the synchronized first modality sample information and second modality sample information;

[0141] Constructing a negative sample according to asynchronous first modality sample information and second modality sample information, or constructing a negative sample according to heterogeneous first modality sample information and second modality sample information;

[0142] The original recognition model is trained according to the constructed positive samples and negative samples to generate an object recognition model.

[0143] In the embodiment of the present application, the acquisition module 301 first acquires the information to be identified, then the extraction module 302 extracts features from the information to be identified to obtain object feature information, and then the determination module 303 determines the target identification object information based on the object feature information.

[0144] In summary, the embodiments of the present application obtain object feature information by extracting features of information to be identified, and determine target identification object information based on the object feature information, thereby improving the accuracy of target identification object identification.

[0145] In addition, the present application also provides a computer device, such as Fig.13As shown, it shows a schematic diagram of the structure of the computer device involved in this application, specifically:

[0146] The computer device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art will appreciate that Fig.13 The computer device structure shown in the figure does not constitute a limitation on the computer device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. Among them:

[0147] The processor 401 is the control center of the computer device, and uses various interfaces and lines to connect various parts of the entire computer device. By running or executing software programs and / or modules stored in the memory 402, and calling data stored in the memory 402, the processor 401 performs various functions of the computer device and processes data. Optionally, the processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 401.

[0148] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0149] The computer device also includes a power supply 403 for supplying power to various components. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 403 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0150] The computer device may further include an input unit 404, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0151] Although not shown, the computer device may also include a display unit, etc., which will not be described in detail here. Specifically in this embodiment, the processor 401 in the computer device will load the executable file corresponding to the process of one or more application programs into the memory 402 according to the following instructions, and the processor 401 will run the application program stored in the memory 402, thereby implementing the steps in any target recognition method provided in this application, specifically:

[0152] The embodiment of the present application obtains information to be identified, performs feature extraction on the information to be identified, obtains object feature information, and determines target identification object information based on the object feature information.

[0153] Among them, the embodiment of the present application obtains object feature information by extracting features of the information to be identified, and determines target identification object information based on the object feature information, thereby improving the accuracy of target identification object identification.

[0154] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0155] To this end, the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in any target recognition method provided in the present application are implemented.

[0156] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0157] Since the instructions stored in the computer-readable storage medium can execute the steps in any target recognition provided in the present application, the beneficial effects that can be achieved by any target recognition method provided in the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0158] The above is a detailed introduction to a target recognition method, system, computer device and computer-readable storage medium based on multimodal information provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

[0159] It should be noted that in the specific implementation of the present application, related data such as multimodal information, posture information and timbre information of the conversation scene, when the above embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

Claims

1. A method, characterized in that include: Obtaining information to be identified; Extracting features from the information to be identified to obtain object feature information; Based on the object feature information, target recognition object information is determined.

2. The method according to claim 1, characterized in that The step of determining target identification object information based on the object feature information includes: Based on the object feature information, determining the object posture information; If the object posture information meets the first preset condition, the object feature information is identified and processed using the object recognition model to obtain target recognition object information; and / or, If the object posture information does not meet the first preset condition, feature analysis and recognition processing is performed on the object feature information to obtain the target recognition object information.

3. The method according to claim 2, characterized in that If the object posture information does not meet the first preset condition, performing feature analysis and recognition processing on the object feature information to obtain the target recognition object information includes: If the object posture information does not satisfy the first preset condition, determining the object timbre information based on the object feature information; and / or, If the object timbre information and the historical timbre information of the previously identified historical object information meet the second preset condition, the object feature information is identified and processed using the object recognition model to obtain the target identification object information; and / or, If the object timbre information and the historical timbre information of the previously recognized historical object information do not meet the second preset condition, the target recognition object information is determined according to the historical object information.

4. The method according to claim 2, characterized in that: The object recognition model includes a first encoding module, a second encoding module, a fusion module and a timing processing module, the input end of the first encoding module is configured to receive the first model input information of the object recognition model, the input end of the second encoding module is configured to receive the second model input information of the object recognition model, the output end of the first encoding module is connected to the input end of the fusion module, the output end of the second encoding module is connected to the input end of the fusion module, the output end of the fusion module is connected to the input end of the timing processing module, and the output end of the timing processing module is configured to output the model output information of the object recognition model.

5. The method according to claim 4, characterized in that The first model input information of the object recognition model is the image information in the information to be recognized, and the second model input information of the object recognition model is the audio information in the information to be recognized; The first encoding module is configured to perform image encoding on the image information to obtain an image embedding representation; The second encoding module is configured to perform audio encoding on the audio information to obtain an audio embedding representation; The fusion module is configured to fuse the image embedding representation and the audio embedding representation to obtain a fused feature representation; The time series processing module is configured to establish a time series representation of the fused feature representation in a time dimension.

6. The method according to claim 5, characterized in that The first encoding module includes a first combination unit, a first pooling unit, a first residual unit, a second combination unit and a second pooling unit in sequence, the input end of the first combination unit is configured to receive the first model input information of the object recognition model, the output end of the first combination unit is connected to the input end of the first pooling unit, the output end of the first pooling unit is connected to the input end of the first residual unit, the output end of the first residual unit is connected to the input end of the second combination unit, the output end of the second combination unit is connected to the input end of the second pooling unit, and the output end of the second pooling unit is connected to the input end of the fusion module; The second encoding module includes a third combination unit, a second residual unit, a fourth combination unit, and a conversion unit, wherein the input end of the third combination unit is configured to receive the second model input information of the object recognition model, the output end of the third combination unit is connected to the input end of the second residual unit, the output end of the second residual unit is connected to the input end of the fourth combination unit, the output end of the fourth combination unit is connected to the input end of the conversion unit, and the output end of the conversion unit is connected to the input end of the fusion module; The fusion module comprises a feature connection unit, the input end of the feature connection unit is respectively connected to the output end of the second pooling unit and the output end of the conversion unit, and the output end of the feature connection unit is connected to the input end of the timing processing module; The timing processing module includes a separation convolution unit, a fifth combination unit and a fully connected unit in sequence, the input end of the separation convolution unit is connected to the output end of the feature connection unit, the output end of the separation convolution unit is connected to the input end of the fifth combination unit, the output end of the fifth combination unit is connected to the input end of the fully connected unit, and the output end of the fully connected unit is configured to output model output information of the object recognition model.

7. The method according to claim 6, characterized in that The combination units corresponding to the first combination unit, the third combination unit, the fourth combination unit and the fifth combination unit include a first convolutional layer, a normalization layer and an activation layer, the second combination unit includes a first convolutional layer and a normalization layer, the residual units corresponding to the first residual unit and the second residual unit include at least one residual sub-unit with the same structure and different parameters, the residual sub-unit includes a sixth combination unit, a seventh combination unit, an eighth combination unit and a feature addition layer, the separation convolution unit includes at least one separation convolution sub-unit with the same structure and the same parameters, and the separation convolution sub-unit includes a second convolutional layer, a normalization layer and an activation layer.

8. The method according to claim 5, characterized in that The image information is a differential image formed by adjacent sequence frames corresponding to the information to be identified.

9. The method according to claim 2, characterized in that: The object recognition model is obtained by executing the following steps: Acquire sample multimodal information, where the sample multimodal information includes first modality sample information and second modality sample information; Constructing a positive sample according to the synchronized first modality sample information and second modality sample information; Constructing a negative sample according to the asynchronous first modality sample information and the second modality sample information, or constructing a negative sample according to the heterogeneous first modality sample information and the second modality sample information; The original recognition model is trained according to the constructed positive samples and negative samples to generate an object recognition model.

10. A system, characterized in that: include: An acquisition module, used for acquiring information to be identified; An extraction module, used for extracting features from the information to be identified to obtain object feature information; A determination module, used to determine target recognition object information based on the object feature information; Preferably, the determination module determines the target identification object information based on the object feature information, including: Based on the object feature information, determining the object posture information; If the object posture information meets the first preset condition, the object feature information is identified and processed using the object recognition model to obtain target recognition object information; and / or, If the object posture information does not meet the first preset condition, performing feature analysis and recognition processing on the object feature information to obtain the target recognition object information; Preferably, the determination module is used to perform feature analysis and recognition processing on the object feature information to obtain the target recognition object information if the object posture information does not meet the first preset condition, including: If the object posture information does not satisfy the first preset condition, determining the object timbre information based on the object feature information; and / or, If the object timbre information and the historical timbre information of the previously identified historical object information meet the second preset condition, the object feature information is identified and processed using the object recognition model to obtain the target identification object information; and / or, If the object timbre information and the historical timbre information of the previously recognized historical object information do not meet the second preset condition, determining the target recognition object information according to the historical object information; Preferably, the object recognition model comprises a first encoding module, a second encoding module, a fusion module and a timing processing module, the input end of the first encoding module is configured to receive the first model input information of the object recognition model, the input end of the second encoding module is configured to receive the second model input information of the object recognition model, the output end of the first encoding module is connected to the input end of the fusion module, the output end of the second encoding module is connected to the input end of the fusion module, the output end of the fusion module is connected to the input end of the timing processing module, and the output end of the timing processing module is configured to output the model output information of the object recognition model; Preferably, the first model input information of the object recognition model is the image information in the information to be recognized, and the second model input information of the object recognition model is the audio information in the information to be recognized; The first encoding module is configured to perform image encoding on the image information to obtain an image embedding representation; The second encoding module is configured to perform audio encoding on the audio information to obtain an audio embedding representation; The fusion module is configured to fuse the image embedding representation and the audio embedding representation to obtain a fused feature representation; The time series processing module is configured to establish a time series representation for the fused feature representation in the time dimension; Preferably, the first encoding module includes a first combination unit, a first pooling unit, a first residual unit, a second combination unit and a second pooling unit in sequence, the input end of the first combination unit is configured to receive the first model input information of the object recognition model, the output end of the first combination unit is connected to the input end of the first pooling unit, the output end of the first pooling unit is connected to the input end of the first residual unit, the output end of the first residual unit is connected to the input end of the second combination unit, the output end of the second combination unit is connected to the input end of the second pooling unit, and the output end of the second pooling unit is connected to the input end of the fusion module; The second encoding module includes a third combination unit, a second residual unit, a fourth combination unit, and a conversion unit, wherein the input end of the third combination unit is configured to receive the second model input information of the object recognition model, the output end of the third combination unit is connected to the input end of the second residual unit, the output end of the second residual unit is connected to the input end of the fourth combination unit, the output end of the fourth combination unit is connected to the input end of the conversion unit, and the output end of the conversion unit is connected to the input end of the fusion module; The fusion module comprises a feature connection unit, the input end of the feature connection unit is respectively connected to the output end of the second pooling unit and the output end of the conversion unit, and the output end of the feature connection unit is connected to the input end of the timing processing module; The timing processing module sequentially comprises a separation convolution unit, a fifth combination unit and a full connection unit, the input end of the separation convolution unit is connected to the output end of the feature connection unit, the output end of the separation convolution unit is connected to the input end of the fifth combination unit, the output end of the fifth combination unit is connected to the input end of the full connection unit, and the output end of the full connection unit is configured to output the model output information of the object recognition model; Preferably, the combination units corresponding to the first combination unit, the third combination unit, the fourth combination unit and the fifth combination unit include a first convolution layer, a normalization layer and an activation layer, the second combination unit includes a first convolution layer and a normalization layer, the residual units corresponding to the first residual unit and the second residual unit include at least one residual sub-unit with the same structure and different parameters, the residual sub-unit includes a sixth combination unit, a seventh combination unit, an eighth combination unit and a feature addition layer, the separation convolution unit includes at least one separation convolution sub-unit with the same structure and the same parameters, and the separation convolution sub-unit includes a second convolution layer, a normalization layer and an activation layer; Preferably, the image information is a differential image formed by adjacent sequence frames corresponding to the information to be identified; Preferably, the object recognition model is obtained by the determination module by executing the following steps: Acquire sample multimodal information, where the sample multimodal information includes first modality sample information and second modality sample information; Constructing a positive sample according to the synchronized first modality sample information and second modality sample information; Constructing a negative sample according to the asynchronous first modality sample information and the second modality sample information, or constructing a negative sample according to the heterogeneous first modality sample information and the second modality sample information; The original recognition model is trained according to the constructed positive samples and negative samples to generate an object recognition model.

11. A device, characterized in that: The device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the method according to any one of claims 1 to 9 when executing the computer program.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images