Object tracking methods, devices, storage media, and robots

Abstract

This application discloses an object tracking method, apparatus, storage medium, and robot, relating to the field of robotics. The method includes: collecting object data of an object to be identified; when the object data includes an object image and voiceprint information, using a first recognition model to extract features from the object image and the voiceprint information and perform cross-modal feature fusion to determine whether the object to be identified is a target object based on the cross-modal fusion features; when the object data includes the object image, using a second recognition model to perform identity recognition based on the object image to determine whether the object to be identified is a target object. This application can improve the accuracy and success rate of object tracking for robots. Furthermore, it can intelligently and dynamically utilize and extract one or more data from sound, face, and body, improving the accuracy and success rate of object tracking for robots in all scenarios.

Description

Technical Field

[0001] This application relates to the field of robotics technology, specifically to an object tracking method, apparatus, storage medium, and robot. Background Technology

[0002] With the continuous development of technology, robots are widely used in scenarios such as home services, making object tracking capabilities particularly important. Object tracking refers to the task of a robot tracking a target object. However, some robots can only move their position and perform tasks according to instructions and cannot perform object tracking. Other robots rely on a single identification method for object tracking, such as facial recognition or voiceprint recognition, to determine whether the target object has been tracked. In special cases such as not being able to collect a face or voice, face occlusion, twins with similar faces, or abnormal or similar voices, misidentification or failure to identify the target object is very likely to occur, resulting in low accuracy and success rate of object tracking. Summary of the Invention

[0003] This application provides an object tracking scheme that can effectively improve the accuracy and success rate of object tracking by robots.

[0004] The embodiments of this application provide the following technical solutions:

[0005] According to one embodiment of this application, an object tracking method includes: acquiring object data of an object to be identified; when the object data includes an object image and voiceprint information, using a first recognition model to perform feature extraction and cross-modal feature fusion on the object image and the voiceprint information, so as to determine whether the object to be identified is a target object based on the cross-modal fusion features; when the object data includes the object image, using a second recognition model to perform identity recognition based on the object image, so as to determine whether the object to be identified is a target object.

[0006] In some embodiments of this application, the step of using a first recognition model to extract features from the object image and the voiceprint information and perform cross-modal feature fusion to determine whether the object to be identified is a target object based on the cross-modal fusion features includes: extracting features from the object image to obtain a first feature; extracting features from the voiceprint information to obtain a second feature; fusing the first feature and the second feature across modal features to obtain the cross-modal fusion feature; matching the cross-modal fusion feature with a preset identity database to obtain a matching result; and determining whether the object to be identified is the target object based on the matching result.

[0007] In some embodiments of this application, the step of fusing the first feature and the second feature across modalities to obtain the cross-modal fused feature includes: mapping the first feature and the second feature to the same semantic space to obtain a third feature and a fourth feature; and aggregating the third feature and the fourth feature based on an attention mechanism to obtain the cross-modal fused feature.

[0008] In some embodiments of this application, the step of using a second recognition model to perform identity recognition based on the object image to determine whether the object to be identified is a target object includes: performing multi-scale feature extraction on the object image to obtain a fifth feature; fusing deep and shallow features of the fifth feature and performing enhancement processing based on a multi-head attention mechanism to obtain a sixth feature; performing multi-attribute identity recognition based on the sixth feature to obtain recognition results for multiple attributes; generating weights for the multiple attributes according to occlusion information in the object image, and performing a weighted summation of the recognition results for the multiple attributes and the weights to obtain a comprehensive recognition result; and determining whether the object to be identified is a target object based on the comprehensive recognition result.

[0009] In some embodiments of this application, the step of performing multi-scale feature extraction on the object image to obtain a fifth feature includes: extracting and fusing features from multiple block images of the object image based on an attention mechanism to obtain shallow features; downsampling the shallow features and processing them based on an attention mechanism to obtain deep features; and obtaining the fifth feature based on the shallow features and the deep features.

[0010] In some embodiments of this application, the second recognition model is trained in the following manner: filtering attributes based on an attribute dataset to obtain selected multiple attributes; and training the second recognition model to be trained using training data based on the selected multiple attributes to obtain the trained second recognition model.

[0011] In some embodiments of this application, the step of filtering attributes based on the attribute dataset to obtain the selected plurality of attributes includes one or more of the following methods: training a corresponding binary classification model for each attribute in the attribute dataset, and filtering the first attribute based on the classification performance of the binary classification model; calculating the dataset entropy of the attribute dataset and calculating the conditional entropy of each attribute in the attribute dataset; calculating the information gain of each attribute based on the dataset entropy and the conditional entropy, and filtering the second attribute based on the information gain of each attribute; obtaining a third attribute specified by the user; the plurality of attributes includes one or more of the first attribute, the second attribute, and the third attribute.

[0012] In some embodiments of this application, the method further includes: when the object data includes a clear face image, performing face recognition based on the clear face image to determine whether the object to be identified is the target object; when the object data does not include the clear face image or the face recognition fails, using the first recognition model or the second recognition model for identity recognition.

[0013] In some embodiments of this application, before obtaining the object data of the object to be identified, the method further includes: receiving object tracking voice; obtaining the target object based on the object tracking voice; and when it is confirmed that the target object is not the sender of the object tracking voice, moving and searching within a predetermined space, and identifying the searched object as the object to be identified.

[0014] According to one embodiment of this application, an object tracking device includes: an acquisition module, configured to: acquire object data of an object to be identified; a first identification module, configured to: when the object data includes an object image and voiceprint information, use a first identification model to perform feature extraction and cross-modal feature fusion on the object image and the voiceprint information, so as to determine whether the object to be identified is a target object based on the cross-modal fusion features; and a second identification module, configured to: when the object data includes the object image, use a second identification model to perform identity recognition based on the object image, so as to determine whether the object to be identified is a target object.

[0015] In some embodiments of this application, when the first recognition model is used to extract features from the object image and the voiceprint information and perform cross-modal feature fusion to determine whether the object to be identified is the target object based on the cross-modal fusion features, the first recognition module can be used to: extract features from the object image to obtain a first feature; extract features from the voiceprint information to obtain a second feature; perform cross-modal feature fusion of the first feature and the second feature to obtain the cross-modal fusion feature; match the cross-modal fusion feature with a preset identity database to obtain a matching result; and determine whether the object to be identified is the target object based on the matching result.

[0016] In some embodiments of this application, when fusing the first feature and the second feature across modalities to obtain the cross-modal fused feature, the first recognition module may be used to: map the first feature and the second feature to the same semantic space to obtain a third feature and a fourth feature; and aggregate the third feature and the fourth feature based on an attention mechanism to obtain the cross-modal fused feature.

[0017] In some embodiments of this application, when the second recognition model is used to perform identity recognition based on the object image to determine whether the object to be identified is a target object, the second recognition module can be used to: extract multi-scale features from the object image to obtain a fifth feature; fuse the fifth feature with deep and shallow features and perform enhancement processing based on a multi-head attention mechanism to obtain a sixth feature; perform multi-attribute identity recognition based on the sixth feature to obtain recognition results for multiple attributes; generate weights for the multiple attributes according to the occlusion information in the object image, and perform a weighted summation of the recognition results for the multiple attributes and the weights to obtain a comprehensive recognition result; and determine whether the object to be identified is a target object based on the comprehensive recognition result.

[0018] In some embodiments of this application, when performing multi-scale feature extraction on the object image to obtain the fifth feature, the second recognition module can be used to: extract and fuse features from multiple block images of the object image based on an attention mechanism to obtain shallow features; downsample the shallow features and process them based on an attention mechanism to obtain deep features; and obtain the fifth feature based on the shallow features and the deep features.

[0019] In some embodiments of this application, the second recognition model is trained in the following manner: filtering attributes based on an attribute dataset to obtain selected multiple attributes; and training the second recognition model to be trained using training data based on the selected multiple attributes to obtain the trained second recognition model.

[0020] In some embodiments of this application, the step of filtering attributes based on the attribute dataset to obtain the selected plurality of attributes includes one or more of the following methods: training a corresponding binary classification model for each attribute in the attribute dataset, and filtering the first attribute based on the classification performance of the binary classification model; calculating the dataset entropy of the attribute dataset and calculating the conditional entropy of each attribute in the attribute dataset; calculating the information gain of each attribute based on the dataset entropy and the conditional entropy, and filtering the second attribute based on the information gain of each attribute; obtaining a third attribute specified by the user; the plurality of attributes includes one or more of the first attribute, the second attribute, and the third attribute.

[0021] In some embodiments of this application, the device further includes a third identification module for: when the object data includes a clear face image, performing face recognition based on the clear face image to determine whether the object to be identified is the target object; when the object data does not include the clear face image or the face recognition fails, using the first identification model or the second identification model for identity recognition.

[0022] In some embodiments of this application, before acquiring the object data of the object to be identified, the device further includes an instruction parsing module for: receiving object tracking voice; obtaining the target object based on the object tracking voice; and, when confirming that the target object is not the sender of the object tracking voice, moving and searching within a predetermined space, and identifying the searched object as the object to be identified.

[0023] According to another embodiment of this application, a storage medium stores a computer program thereon, which, when executed by a robot's processor, causes the robot to perform the methods described in the embodiments of this application.

[0024] According to another embodiment of this application, a robot may include: a memory storing a computer program; and a processor reading the computer program stored in the memory to execute the methods described in the embodiments of this application.

[0025] According to another embodiment of this application, a computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A robot's processor reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the robot to perform the methods provided in the various optional implementations described in the embodiments of this application.

[0026] In this embodiment of the application, object data of the object to be identified is obtained; when the object data includes object image and voiceprint information, a first recognition model is used to extract features and fuse cross-modal features of the object image and the voiceprint information, so as to determine whether the object to be identified is the target object based on the cross-modal fusion features; when the object data includes the object image, a second recognition model is used to perform identity recognition based on the object image, so as to determine whether the object to be identified is the target object.

[0027] In this embodiment of the application, firstly, even when the object to be identified has a face obscured, it is very easy to successfully acquire an image of the object that does not contain a face or has an unclear face, but at least contains other body parts. Furthermore, when voiceprint information is not acquired, a second recognition model can be used to perform identity recognition based on the object image. And when voiceprint information is further acquired, using a first recognition model combined with the object image and voiceprint information can achieve more accurate identity recognition. Thus, in various special circumstances, it can generally ensure that the robot successfully and accurately tracks the target object, improving the robot's object tracking accuracy and success rate. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 A flowchart of an object tracking method according to an embodiment of this application is shown.

[0030] Figure 2 A schematic diagram of a robot according to one embodiment of this application is shown.

[0031] Figure 3 A flowchart of an object search process according to an embodiment of this application is shown.

[0032] Figure 4 A flowchart of an object tracking method according to another embodiment of this application is shown.

[0033] Figure 5 An identity recognition flowchart according to an embodiment of this application is shown.

[0034] Figure 6 A flowchart illustrating an identity recognition process according to another embodiment of this application is shown.

[0035] Figure 7 A flowchart of attribute filtering according to an embodiment of this application is shown.

[0036] Figure 8 A block diagram of an object tracking device according to an embodiment of this application is shown.

[0037] Figure 9 A block diagram of a robot according to one embodiment of this application is shown. Detailed Implementation

[0038] The present disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the embodiments provided herein are merely illustrative of the present disclosure and are not intended to limit the present disclosure. Furthermore, the embodiments provided below are some embodiments for implementing the present disclosure, and not all embodiments for implementing the present disclosure. Unless otherwise specified, the technical solutions described in the embodiments of the present disclosure can be implemented in any combination.

[0039] It should be noted that, in the embodiments of this disclosure, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a method or apparatus that includes a list of elements includes not only the elements expressly described, but also other elements not expressly listed, or elements inherent to implementing the method or apparatus. Without further limitations, an element defined by the phrase "comprising a..." does not exclude the presence of other related elements (e.g., steps in the method or units in the apparatus, such as portions of circuitry, processors, programs, or software, etc.) in the method or apparatus that includes that element.

[0040] For example, the object tracking method provided in this disclosure includes a series of steps, but the object tracking method provided in this disclosure is not limited to the steps described. Similarly, the object tracking device provided in this disclosure includes a series of units, but the device provided in this disclosure is not limited to the units explicitly described, but may also include units that need to be set up for obtaining relevant information or processing based on the information.

[0041] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of this disclosure.

[0042] It is understood that in the specific implementation of this application, relevant data is involved. When the embodiments in this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0043] With the continuous development of technology, robots are widely used in scenarios such as home services, making object tracking capabilities particularly important. Object tracking refers to the task of a robot tracking a target object. However, some robots can only move their position and perform tasks according to instructions and cannot perform object tracking. Other robots rely on a single identification method for object tracking, such as facial recognition or voiceprint recognition, to determine whether the target object has been tracked. In special cases such as not being able to collect a face or voice, face occlusion, twins with similar faces, or abnormal or similar voices, misidentification or failure to identify the target object is very likely to occur, resulting in low accuracy and success rate of object tracking.

[0044] To address these issues, this application provides an object tracking solution that can effectively improve the accuracy and success rate of object tracking for robots.

[0045] The following is a detailed description of relevant embodiments of the object tracking scheme provided in this application.

[0046] Figure 1 A flowchart illustrating an object tracking method according to an embodiment of this application is shown. This object tracking method is applicable to a robot, which may include at least a memory and a processor. The processor in the robot can read computer programs stored in the memory to execute the methods of various embodiments of this application.

[0047] like Figure 1 As shown, the object tracking method may include steps S110 to S130.

[0048] Step S110: Obtain object data of the object to be identified;

[0049] Step S120: When the object data includes object image and voiceprint information, the first recognition model is used to extract features and fuse cross-modal features of the object image and voiceprint information, so as to determine whether the object to be identified is the target object based on the cross-modal fusion features.

[0050] Step S130: When the object data includes an object image, the second recognition model is used to perform identity recognition based on the object image to determine whether the object to be identified is the target object.

[0051] After receiving an instruction to track a target object, the robot can move autonomously to find the target object, using objects encountered along the way as objects to be identified. While ensuring obstacle avoidance, the robot can follow the object to be identified, capturing images of the object from multiple angles using a camera within a predetermined range (e.g., within 1 meter), and collecting the object's sound using a microphone, etc., to obtain object data based on the collected images and sound. The robot in this application can be a humanoid robot or a roller-skating robot, etc., and this application does not specifically limit the robot's external shape or structure.

[0052] An example robot such as Figure 2 As shown, the robot 200 may include a small host 210 and a chassis 220. Cameras may be installed on both the small host 210 and the chassis 220. The small host 210 can rotate 360 ​​degrees on the chassis. An algorithm module 221, a sound pickup module 222 (such as a microphone), and a detection module 223 (such as an infrared sensor, lidar, etc.) may also be installed on the chassis 220. The algorithm module 221 can control the robot to execute the steps of the various embodiments of this application.

[0053] The object image can be an image of the object to be identified. It can be an unsegmented whole image, or it can be a segmented image of multiple parts (e.g., a whole image can be segmented into images of the upper body, head, feet, legs, etc., using an object detection network or image segmentation network). Voiceprint information can be extracted from the sound of the object to be identified, such as fundamental frequency, formants, and spectral features. The robot can use a sound recognition model to analyze the location of the sound source and determine whether the sound is that of the object to be identified.

[0054] The target object can be a person, an animal, or another robot. For example, the user could give instructions such as "go find Mom," "follow me," or "go find the dog." The robot can then determine the target object to be tracked based on these instructions, such as Mom, itself, or the dog. During the tracking process, when the robot detects an object to be identified (such as a person or animal that is similar to the target object), it can collect object data for that object.

[0055] Furthermore, pre-trained first and second recognition models are deployed. These models do not rely on facial images for face recognition to identify the object. That is, even if the object image does not contain a face or the face is unclear, the first recognition model can still perform identification based on the object image and voiceprint information, while the second recognition model can perform identification based solely on the object image. The first and second recognition models can be deployed locally on the robot, or they can be deployed on a server (such as a cloud or other terminal that can establish a connection with the robot). The first and second recognition models are pre-trained deep learning models.

[0056] When the object data includes object image and voiceprint information, the object image and voiceprint information are input together into the first recognition model (such as inputting to the first recognition model local to the deployed robot, or remotely sent to the server and then input to the first recognition model deployed on the server). The first recognition model can perform feature extraction and cross-modal feature fusion on the object image and voiceprint information, and determine whether the object to be identified is the target object based on the cross-modal fusion features. That is, it combines the data of the two modalities of object image and voiceprint information to perform cross-modal comprehensive reasoning (that is, to perform identity recognition) and accurately determine whether the object to be identified is the target object that needs to be tracked.

[0057] When the object data does not include voiceprint information (such as the object's voice not being collected, or the voiceprint information not being extracted due to weak sound), but includes an object image, the object image is input into the second recognition model (such as inputting it into the second recognition model local to the deployed robot, or remotely sending it to the server and then inputting it into the second recognition model deployed on the server). The second recognition model can perform reasoning (i.e., identity recognition) based on the object image to determine whether the object to be identified is the target object that needs to be tracked.

[0058] When the robot collects the image and voiceprint information of the object, it can use the first recognition model to perform identity recognition through cross-modal integrated reasoning, thereby more accurately determining whether the object to be identified is the target object that needs to be tracked; when the robot does not collect voiceprint information, it can also use the second recognition model to perform identity recognition and determine whether the object to be identified is the target object that needs to be tracked.

[0059] The tracking process ends when the object to be identified is the target object to be tracked. If the object to be identified is not the target object to be tracked, the robot continues to search for the next object to be identified until it is confirmed to be the target object to be tracked.

[0060] In summary, using the method described in this embodiment, firstly, even when the object to be identified has a face obscured, it is very easy to successfully acquire an image of the object that does not contain a face or has an unclear face, but at least contains other body parts. Furthermore, when voiceprint information is not acquired, a second recognition model can be used for identity recognition based on the object image. And when voiceprint information is further acquired, using a first recognition model combined with the object image and voiceprint information can achieve more accurate identity recognition. Thus, in various special circumstances, it can generally ensure that the robot successfully and accurately tracks the target object, improving the robot's object tracking accuracy and success rate.

[0061] The following description Figure 1 Further optional specific embodiments are provided for each step performed during object tracking in the example implementation.

[0062] See Figure 3 In one embodiment, before obtaining the object data of the object to be identified in step S110, the method may further include: step S101, receiving object tracking voice; step S102, obtaining the target object based on the object tracking voice; and step S103, when it is confirmed that the target object is not the sender of the object tracking voice, searching within a predetermined space and identifying the searched object as the object to be identified.

[0063] The robot receives object-tracking speech from a user (the sender), which is the speech used to track the target object, such as "Little T, go find your mother." The robot can use a voice recognition model to parse the object-tracking speech and determine the indicated target object (such as the mother). Furthermore, the voice recognition model can also parse the object-tracking speech to determine the location of the sender's sound source. The voice recognition model can be deployed locally on the robot or on a server. In one example, the voice recognition model is a Large LLM model or other speech processing model.

[0064] Furthermore, the robot can determine whether the target object is the source of the object tracking voice. If it confirms that the target object is not the source of the voice tracking, the robot can initiate a global exploration mode. Within a predetermined space (such as a home area), the robot autonomously moves and searches using localization mapping, path planning, and active obstacle avoidance, identifying the found objects as targets for identification. Then, it collects the object data of these targets to determine if they are indeed the target object. Subsequently, the robot can autonomously analyze commands and actively search for the target object, without requiring the target object to actively appear in the robot's field of vision.

[0065] Furthermore, after identifying the target object based on the object tracking speech, the robot can also execute step S104: if the target object is determined to be the originator of the object tracking speech, the robot can move to the vicinity of the originator based on the sound source location given by the sound recognition model. Specifically, the robot can extract the voiceprint information of the object tracking speech, and if the voiceprint information belongs to the target object, determine that the target object is the originator of the object tracking speech. Alternatively, the robot can determine that the target object is the originator of the object tracking speech when the descriptive word of the target object in the object tracking speech is a predetermined word (e.g., the descriptive word is "I").

[0066] In addition, during the robot's mobile search, the methods for determining the searched object may include, but are not limited to: detecting an object of the same type as the target object through infrared or lidar; or, using a target detection network (such as yolov8n, You Only Look Once version 8 Nano) to perform target detection on the image captured by the camera and detect the presence of an object of the same type as the target object in the image.

[0067] For further details, please refer to [link / reference]. Figure 4In one embodiment, after obtaining the object data of the object to be identified in step S110, the method may further include: step S112, when the object data includes a clear face image, performing face recognition based on the clear face image to determine whether the object to be identified is the target object; step S114, when the object data does not include a clear face image or face recognition fails, determining to use a first recognition model or a second recognition model for identity recognition.

[0068] When a face image is segmented from a photographic image using an object detection network or image segmentation network, and this face image is a clear face image with a resolution higher than a predetermined resolution, a face recognition model can be used to perform face recognition based on this clear face image to determine whether the object to be identified is the target object. However, if the object data does not include a clear face image (e.g., no face image is segmented or the resolution of the segmented face image is lower than a predetermined resolution) or if the aforementioned face recognition fails, then a first recognition model or a second recognition model is used for identity verification.

[0069] Therefore, when a clear facial image is available, facial recognition can be used to efficiently and accurately determine whether the object to be identified is the target object. If a clear facial image is not available, or if facial recognition fails as described above, a first or second recognition model can be used for identification. This triple recognition further improves the robot's object tracking success rate. Furthermore, it can intelligently and dynamically utilize and extract one or more data points from sound, face (facial image), and body (object image), improving the robot's object tracking accuracy and success rate across all scenarios.

[0070] See Figure 5 In one embodiment, step S120, using a first recognition model to extract features from the object image and voiceprint information and perform cross-modal feature fusion to determine whether the object to be identified is the target object based on the cross-modal fusion features, may include: step S121, extracting features from the object image to obtain a first feature; step S122, extracting features from the voiceprint information to obtain a second feature; step S123, fusing the first feature and the second feature across modal features to obtain a cross-modal fusion feature; step S124, matching the cross-modal fusion feature with a preset identity database to obtain a matching result; and determining whether the object to be identified is the target object based on the matching result.

[0071] After inputting the object image and voiceprint information into the first recognition model, the first feature can be obtained by extracting features from the object image through the first feature extraction layer (such as a convolutional layer), and the second feature can be obtained by extracting features from the voiceprint information through the second feature extraction layer (such as a convolutional layer). Furthermore, the first feature and the second feature can be fused across modalities through a fusion layer to obtain a multimodal fusion feature.

[0072] The preset identity database allows setting preset features corresponding to target objects. These preset features can be obtained by fusing cross-modal features of the extracted image and voiceprint of the target object. The obtained cross-modal fused features are then matched with the preset features of the target object in the preset identity database to obtain a matching result. Based on the matching result, it can be accurately determined whether the object to be identified is the target object. For example, if the similarity between the fused features and the preset features of the target object is greater than a preset similarity threshold, then the object to be identified is determined to be the target object; otherwise, the object to be identified is determined not to be the target object.

[0073] The features in this application can be in the form of feature maps or feature vectors, and the specific form can be set according to the actual situation.

[0074] Optionally, in one embodiment, the first feature and the second feature are fused across modal features to obtain cross-modal fused features. This can be achieved by mapping the first feature and the second feature to the same semantic space and then fusing them by concatenation, addition, or multiplication to obtain cross-modal fused features.

[0075] Furthermore, in one embodiment, fusing the first feature and the second feature across modalities to obtain cross-modal fused features may include: mapping the first feature and the second feature to the same semantic space to obtain a third feature and a fourth feature; and aggregating the third feature and the fourth feature based on an attention mechanism to obtain cross-modal fused features.

[0076] In this embodiment, the first feature and the second feature are mapped to the same semantic space to obtain a third feature corresponding to the first feature and a fourth feature corresponding to the second feature. Then, the third feature and the fourth feature are aggregated across modally based on an attention mechanism to obtain a cross-modal fusion feature. The attention mechanism can better achieve cross-modal fusion, and the fusion feature obtained in this way can further improve the recognition accuracy when used for identity recognition.

[0077] In one specific approach, the third and fourth features are aggregated based on an attention mechanism to obtain cross-modal fusion features. This may include: concatenating the third and fourth features to obtain concatenated features; generating a third query feature, a third key feature, and a third value feature based on the concatenated features; and performing operations on the third query feature, the third key feature, and the third value feature based on a self-attention formula to obtain cross-modal fusion features.

[0078] Specifically, the third feature is T3, the fourth feature is T4, and the concatenation feature X = [T3, T4], where [T3, T4] is the row concatenation of T3 and T4; then, the third query feature Q3 = X * W 3-1 The third bond feature K3 = X * W 3-2 The third-valued feature V3 = X * W 3-3 W 3-1 W 3-2 and W 3-3 This is the projection matrix for the attention mechanism.

[0079] Then, the self-attention formula H1=Attn(Q3,K3,V3)=softmax(Q3*K3) is used. T / ) V3, by performing operations on the third query feature Q3, the third key feature K3, and the third value feature V3, we can obtain the cross-modal fusion feature H1, where d k Here, Attn is the key vector dimension for attention, and Attn() refers to the self-attention operation. This implementation effectively fuses features from both image and voiceprint modalities.

[0080] Furthermore, in one specific approach, aggregating the third and fourth features based on an attention mechanism to obtain a cross-modal fusion feature may include: generating a first query feature based on the third feature, and generating a first key feature and a first value feature based on the fourth feature; calculating a first attention weight based on the first query feature, the first key feature, and the first value feature; multiplying the first attention weight by the first value feature to obtain a first attention fusion feature; generating a second query feature based on the fourth feature, and generating a second key feature and a second value feature based on the third feature; calculating a second attention weight based on the second query feature, the second key feature, and the second value feature; multiplying the second attention weight by the second value feature to obtain a second attention fusion feature; then, adding the first attention fusion feature to the third feature to obtain a first superimposed feature; adding the second attention fusion feature to the fourth feature to obtain a second superimposed feature; and fusing the first superimposed feature and the second superimposed feature (e.g., through pooling or concatenation) to obtain the cross-modal fusion feature.

[0081] Specifically, the third feature is T3, the fourth feature is T4, and the first query feature is Q1 = T3 * W. 1-1The first key feature K1 = T4 * W 1-2 The first characteristic V1 = T4 * W 1-3 Among them, W 1-1 W 1-2 and W 1-3 This is the projection matrix for the attention mechanism. The attention formula Wa1 = softmax(Q1 * K1) is used. T / V1 can be calculated based on the first query feature Q1, the first key feature K1, and the first value feature V1 to obtain the first attention weight Wa1. Multiplying the first attention weight Wa1 by the first value feature V1 yields the first attention fusion feature h1.

[0082] The second query feature Q2 = T4 * W 2-1 The second bond feature K2 = T3 * W 2-2 The second-valued feature V2 = T3 * W 2-3 Among them, W 2-1 W 2-2 and W 2-3 This is the projection matrix for the attention mechanism. The attention formula Wa2 = softmax(Q2 * K2) is used. T / V2 can be used to calculate the first attention weight Wa2 based on the second query feature Q2, the second key feature K2, and the second value feature V2. Multiplying the second attention weight Wa2 by the second value feature V2 yields the second attention fusion feature h2.

[0083] Then, the first attention fusion feature h1 is added to the third feature T3 to obtain the first superimposed feature D1; the second attention fusion feature h2 is added to the fourth feature T4 to obtain the second superimposed feature D2; the first superimposed feature D1 and the second superimposed feature D2 are fused (e.g., through pooling or splicing) to obtain the cross-modal fusion feature H2. The applicant found that with this implementation method, the features of the image and voiceprint modalities can be fused more effectively.

[0084] See Figure 6In one embodiment, step S130, using a second recognition model to perform identity recognition based on an object image to determine whether the object to be identified is the target object, may include: step S131, performing multi-scale feature extraction on the object image to obtain a fifth feature; step S132, fusing deep and shallow features of the fifth feature and performing enhancement processing based on a multi-head attention mechanism to obtain a sixth feature; step S133, performing multi-attribute identity recognition based on the sixth feature to obtain recognition results for multiple attributes; step S134, generating weights for multiple attributes based on occlusion information in the object image, and weighting and summing the recognition results of multiple attributes with the weights to obtain a comprehensive recognition result; step S135, determining whether the object to be identified is the target object based on the comprehensive recognition result.

[0085] In the second recognition model, a fifth feature can be obtained by multi-scale feature extraction of the object image through a feature extraction layer. The fifth feature can include shallow features and deep features. A fusion layer can be used to fuse the shallow and deep features in the fifth feature to obtain a shallow-deep fused feature (for example, shallow and deep features can be fused through a feature pyramid network). The shallow-deep fused feature is then enhanced based on a multi-head attention mechanism to obtain a sixth feature (i.e., the shallow-deep fused feature is enhanced by applying multi-head attention to obtain the sixth feature, where the number of heads in the multi-head attention mechanism can be set according to the actual situation).

[0086] Furthermore, a multi-attribute identification layer (such as an MLP layer) can be used to perform multi-attribute identity recognition based on the sixth feature to obtain the recognition results of multiple attributes. These multiple attributes may include, but are not limited to, hair color, hairstyle, clothing type, clothing color, gender, age, etc. The recognition result of each attribute can be the probability that the object to be identified is the target object as reflected by each attribute.

[0087] Furthermore, in the second recognition model, a dynamic weighting layer can generate weights for multiple attributes based on occlusion information in the object image. The recognition results of these multiple attributes are then weighted and summed to obtain a comprehensive recognition result, which can be a probability-weighted sum. Subsequently, when the probability-weighted sum exceeds a preset probability threshold, the object to be recognized can be accurately identified as the target object; otherwise, it is determined that the object to be recognized is not the target object. The dynamic weighting layer assigns different weights to the multiple attributes based on different occlusion information.

[0088] The occlusion information can be obtained from the object image detected by the second recognition model; alternatively, the occlusion information can be obtained from the object image detected by a preset segmentation network during object data acquisition. In this case, the occlusion information can be input into the second recognition model along with the object image. The occlusion information can describe the occluded area, part, or degree of occlusion in the object image.

[0089] Furthermore, in one embodiment, multi-scale feature extraction is performed on the object image to obtain a fifth feature, including: extracting and fusing features from multiple block images of the object image based on an attention mechanism to obtain shallow features; downsampling the shallow features and processing them based on an attention mechanism to obtain deep features; and obtaining the fifth feature based on the shallow features and the deep features.

[0090] The feature extraction layer is used for multi-scale feature extraction. In this embodiment, multi-scale feature extraction can include a shallow feature extraction layer with an attention mechanism and a deep feature extraction layer with an attention mechanism. The shallow feature extraction layer can divide the object image into blocks, extract features based on the attention mechanism, and fuse them to obtain shallow features; the deep feature extraction layer can downsample the shallow features and process them based on the attention mechanism to obtain deep features; finally, the output is a fifth feature including the shallow features and the deep features.

[0091] The fifth feature is obtained by introducing an attention mechanism to extract features at multiple scales. The fifth feature obtained in this way is used in the above embodiments to further improve the accuracy of object recognition.

[0092] Specifically, the shallow features are obtained by extracting and fusing features from multiple blocks of the object image based on an attention mechanism. This can be achieved by: performing convolution processing on each block of the object image to obtain convolution features for each block; processing the convolution features of each block separately through an attention mechanism to obtain block attention features for each block; and then fusing the block attention features of multiple blocks (e.g., by concatenation or summation) to obtain shallow features.

[0093] Specifically, when the object image is an unsegmented whole image, the second recognition model can segment the object image to obtain multiple block images; when the object image is a segmented multiple block images, the above processing can be performed directly based on the input multiple block images.

[0094] Specifically, shallow features are downsampled and processed based on an attention mechanism to obtain deep features. This can be achieved by: downsampling shallow features through convolution or pooling operations to obtain downsampled features; and then processing the downsampled features through an attention mechanism to obtain deep features.

[0095] In the embodiments of this application, the first recognition model and the second recognition model are pre-trained deep learning models, which can be used to implement the steps of the aforementioned embodiments. For example, in one example, the first recognition model can be a fine-tuned InternVL2 (Intern Vision Language) model, and in another example, the second recognition model can be a fine-tuned ALM-A (Multi-Scale Attribute-Specific Localization Attention) model. The first recognition model and the second recognition model can also be other optional deep learning models. The specific training methods for the first recognition model and the second recognition model can follow conventional model training methods, and this application does not impose any special limitations on them.

[0096] Furthermore, in one embodiment, the second recognition model is trained as follows: attributes are filtered according to the attribute dataset to obtain multiple selected attributes; the second recognition model to be trained is trained using training data based on the multiple selected attributes to obtain the trained second recognition model.

[0097] Attribute datasets can include, but are not limited to, attributes such as hair color, hairstyle, clothing type, clothing color, gender, and age. For example, the attribute dataset can be obtained by taking the union of the attributes from three public datasets: PETA, PA100K, and RAP. Attribute filtering based on this dataset allows users to select multiple attributes that meet their needs or have practical effects on object tracking.

[0098] Then, based on the selected multiple attributes, the training data is used to train the second recognition model to be trained, and the trained second recognition model is obtained. The trained second recognition model can then perform multi-attribute identity recognition to obtain the recognition results of the selected multiple attributes, and generate the weights of multiple attributes based on the occlusion information in the object image.

[0099] Furthermore, in one embodiment, filtering attributes based on the attribute dataset to obtain selected attributes may include one or more of the following methods:

[0100] The first method involves training a corresponding binary classification model for each attribute in the attribute dataset, and then selecting the first attribute based on the classification performance of the binary classification model.

[0101] The second method involves calculating the dataset entropy of the attribute dataset and the conditional entropy of each attribute in the attribute dataset; then calculating the information gain of each attribute based on the dataset entropy and conditional entropy, and finally filtering the second attribute based on the information gain of each attribute.

[0102] The third method is to obtain a third attribute specified by the user; multiple attributes include one or more of the first, second, and third attributes.

[0103] In the first approach, a corresponding binary classification model is trained for each attribute in the attribute dataset. The binary classification model can be used to classify images of the corresponding attributes. The classification performance of the binary classification model (such as recall and accuracy) is statistically analyzed. The attributes corresponding to multiple binary classification models whose classification performance ranks ahead of a predetermined name can be selected as the first attribute. Alternatively, the attributes corresponding to multiple binary classification models whose classification performance is higher than a predetermined condition (such as accuracy higher than a predetermined accuracy) can be selected as the first attribute.

[0104] In the second approach, the entropy of the entire attribute dataset and the conditional entropy of each attribute in the attribute dataset are calculated. Then, the conditional entropy of each attribute is subtracted from the dataset entropy to obtain the information gain of each attribute. Furthermore, attributes with information gains higher than a predetermined gain can be selected as the second attribute.

[0105] In the third method, users can specify attributes according to their needs, and the user-specified attributes will be used as the third attributes; for example, users can specify attributes that do not exist in the first and second attributes in the first two methods as the third attributes.

[0106] One or more of the three methods described above are used to filter attributes, and the selected attributes include one or more of the first attribute, the second attribute, and the third attribute.

[0107] See Figure 7 In one specific embodiment, the first attribute, second attribute, and third attribute are obtained through the three methods described above in steps S310, S320, and S330, respectively. In step S340, the selected multiple attributes are determined to include the first attribute, second attribute, and third attribute. In this way, multiple attributes that meet user needs and have practical effects on object tracking can be selected to train the second recognition model, further improving the identity recognition performance of the second recognition model.

[0108] To facilitate better implementation of the object tracking method provided in this application, this application also provides an object tracking device based on the above-described object tracking method. The meanings of the terms used are the same as in the object tracking method described above, and specific implementation details can be found in the descriptions within the method embodiments. Figure 8 A block diagram of an object tracking device according to an embodiment of this application is shown.

[0109] like Figure 8As shown, the object tracking device 400 may include: an acquisition module 410 for acquiring object data of an object to be identified; a first identification module 420 for extracting features and fusing cross-modal features from the object image and the voiceprint information using a first identification model when the object data includes an object image and voiceprint information, so as to determine whether the object to be identified is a target object based on the cross-modal fusion features; and a second identification module 430 for identifying the object based on the object image when the object data includes the object image, so as to determine whether the object to be identified is a target object.

[0110] In some embodiments of this application, when the first recognition model is used to extract features from the object image and the voiceprint information and perform cross-modal feature fusion to determine whether the object to be identified is the target object based on the cross-modal fusion features, the first recognition module can be used to: extract features from the object image to obtain a first feature; extract features from the voiceprint information to obtain a second feature; perform cross-modal feature fusion of the first feature and the second feature to obtain the cross-modal fusion feature; match the cross-modal fusion feature with a preset identity database to obtain a matching result; and determine whether the object to be identified is the target object based on the matching result.

[0111] In some embodiments of this application, when fusing the first feature and the second feature across modalities to obtain the cross-modal fused feature, the first recognition module may be used to: map the first feature and the second feature to the same semantic space to obtain a third feature and a fourth feature; and aggregate the third feature and the fourth feature based on an attention mechanism to obtain the cross-modal fused feature.

[0112] In some embodiments of this application, when the second recognition model is used to perform identity recognition based on the object image to determine whether the object to be identified is a target object, the second recognition module can be used to: extract multi-scale features from the object image to obtain a fifth feature; fuse the fifth feature with deep and shallow features and perform enhancement processing based on a multi-head attention mechanism to obtain a sixth feature; perform multi-attribute identity recognition based on the sixth feature to obtain recognition results for multiple attributes; generate weights for the multiple attributes according to the occlusion information in the object image, and perform a weighted summation of the recognition results for the multiple attributes and the weights to obtain a comprehensive recognition result; and determine whether the object to be identified is a target object based on the comprehensive recognition result.

[0113] In some embodiments of this application, when performing multi-scale feature extraction on the object image to obtain the fifth feature, the second recognition module can be used to: extract and fuse features from multiple block images of the object image based on an attention mechanism to obtain shallow features; downsample the shallow features and process them based on an attention mechanism to obtain deep features; and obtain the fifth feature based on the shallow features and the deep features.

[0114] In some embodiments of this application, the second recognition model is trained in the following manner: filtering attributes based on an attribute dataset to obtain selected multiple attributes; and training the second recognition model to be trained using training data based on the selected multiple attributes to obtain the trained second recognition model.

[0115] In some embodiments of this application, the step of filtering attributes based on the attribute dataset to obtain the selected plurality of attributes includes one or more of the following methods: training a corresponding binary classification model for each attribute in the attribute dataset, and filtering the first attribute based on the classification performance of the binary classification model; calculating the dataset entropy of the attribute dataset and calculating the conditional entropy of each attribute in the attribute dataset; calculating the information gain of each attribute based on the dataset entropy and the conditional entropy, and filtering the second attribute based on the information gain of each attribute; obtaining a third attribute specified by the user; the plurality of attributes includes one or more of the first attribute, the second attribute, and the third attribute.

[0116] In some embodiments of this application, the device further includes a third identification module for: when the object data includes a clear face image, performing face recognition based on the clear face image to determine whether the object to be identified is the target object; when the object data does not include the clear face image or the face recognition fails, using the first identification model or the second identification model for identity recognition.

[0117] In some embodiments of this application, before acquiring the object data of the object to be identified, the device further includes an instruction parsing module for: receiving object tracking voice; obtaining the target object based on the object tracking voice; and, when confirming that the target object is not the sender of the object tracking voice, moving and searching within a predetermined space, and identifying the searched object as the object to be identified.

[0118] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0119] Furthermore, embodiments of this application also provide a robot, such as Figure 9 As shown, Figure 9 A block diagram of a robot according to an embodiment of this application is shown, specifically:

[0120] The robot may include components such as a processor 501 with one or more processing cores, a memory 502 with one or more computer-readable storage media, a power supply 503, and an input unit 504. Those skilled in the art will understand that... Figure 9 The robot structure shown does not constitute a limitation on the robot and may include more or fewer parts than shown, or combine certain parts, or have different arrangements of parts. Wherein:

[0121] The processor 501 is the control center of the robot, connecting various parts of the computer device via various interfaces and lines. It executes software programs and / or modules stored in the memory 502, and calls data stored in the memory 502, to perform various functions of the computer device and process data. Optionally, the processor 501 may include one or more processing cores; preferably, the processor 501 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user page, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 501.

[0122] The memory 502 can be used to store software programs and modules. The processor 501 executes various functional applications and data processing by running the software programs and modules stored in the memory 502. The memory 502 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created based on the robot's use, etc. In addition, the memory 502 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 502 may also include a memory controller to provide the processor 501 with access to the memory 502.

[0123] The robot also includes a power supply 503 that supplies power to the various components. Preferably, the power supply 503 can be logically connected to the processor 501 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 503 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0124] The robot may also include an input unit 504, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0125] Although not shown, the robot may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 501 in the robot can load the executable files corresponding to the processes of one or more computer programs into the memory 502 according to the following instructions, and the processor 501 runs the computer programs stored in the memory 502, thereby realizing the various functions in the foregoing embodiments of this application.

[0126] For example, processor 501 can perform the following: acquire object data of the object to be identified; when the object data includes an object image and voiceprint information, use a first recognition model to perform feature extraction and cross-modal feature fusion on the object image and the voiceprint information, so as to determine whether the object to be identified is the target object based on the cross-modal fusion features; when the object data includes the object image, use a second recognition model to perform identity recognition based on the object image, so as to determine whether the object to be identified is the target object.

[0127] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by a computer program, or by a computer program controlling related hardware. The computer program can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0128] Therefore, embodiments of this application also provide a storage medium storing a computer program that can be loaded by a processor to execute the steps in any of the methods provided in embodiments of this application.

[0129] The storage medium can be a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0130] Since the computer program stored in the storage medium can execute the steps of any of the methods provided in the embodiments of this application, the beneficial effects that the methods provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0131] According to another embodiment of this application, a computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A robot's processor reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the robot to perform the methods provided in the various optional implementations described in the embodiments of this application.

[0132] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0133] It should be understood that this application is not limited to the embodiments described above and shown in the accompanying drawings, but various modifications and changes can be made without departing from its scope.

Claims

1. An object tracking method, characterized in that, Applicable to robots, the method includes: Obtain the object data of the object to be identified; When the object data includes a clear facial image, facial recognition is performed based on the clear facial image to determine whether the object to be identified is the target object; When the object data does not include the clear facial image or the facial recognition fails, the first recognition model or the second recognition model is used for identity recognition. When the object data includes object image and voiceprint information, a first recognition model is used to extract features and fuse cross-modal features of the object image and the voiceprint information, so as to determine whether the object to be identified is the target object based on the cross-modal fusion features; When the object data includes the object image, a second recognition model is used to perform identity recognition based on the object image to determine whether the object to be identified is the target object. The step of using a second recognition model to perform identity recognition based on the object image to determine whether the object to be identified is the target object includes: Multi-scale feature extraction is performed on the object image to obtain the fifth feature; The fifth feature is then fused with both shallow and deep features and enhanced using a multi-head attention mechanism to obtain the sixth feature. Based on the sixth feature, multi-attribute identity recognition is performed to obtain recognition results for multiple attributes; The weights of the multiple attributes are generated based on the occlusion information in the object image, and the recognition results of the multiple attributes are weighted and summed with the weights to obtain a comprehensive recognition result. Based on the comprehensive identification results, determine whether the object to be identified is the target object; The step of performing multi-scale feature extraction on the object image to obtain the fifth feature includes: The object image is divided into multiple blocks, and features are extracted and fused based on an attention mechanism to obtain shallow features. The shallow features are downsampled and processed based on an attention mechanism to obtain deep features; The fifth feature is obtained based on the shallow features and the deep features.

2. The method according to claim 1, characterized in that, The step of using a first recognition model to extract features from the object image and the voiceprint information and perform cross-modal feature fusion, so as to determine whether the object to be identified is the target object based on the cross-modal fusion features, includes: Feature extraction is performed on the object image to obtain the first feature; The voiceprint information is used to extract features to obtain a second feature; The first feature and the second feature are fused across modal features to obtain the cross-modal fused feature; The cross-modal fusion features are matched with a preset identity database to obtain the matching results; Based on the matching results, it is determined whether the object to be identified is the target object.

3. The method according to claim 2, characterized in that, The step of fusing the first feature and the second feature across modalities to obtain the cross-modal fused feature includes: The first feature and the second feature are mapped to the same semantic space to obtain the third feature and the fourth feature; The third and fourth features are aggregated based on an attention mechanism to obtain the cross-modal fusion feature.

4. The method according to claim 1, characterized in that, The second recognition model was trained in the following manner: Attributes are filtered based on the attribute dataset to obtain the selected multiple attributes; Based on the selected multiple attributes, the second recognition model to be trained is trained using training data to obtain the trained second recognition model.

5. The method according to claim 4, characterized in that, The step of filtering attributes based on the attribute dataset to obtain the selected multiple attributes includes one or more of the following methods: For each attribute in the attribute dataset, a corresponding binary classification model is trained, and the first attribute is selected based on the classification performance of the binary classification model. Calculate the dataset entropy of the attribute dataset and the conditional entropy of each attribute in the attribute dataset; The information gain of each attribute is calculated based on the dataset entropy and the conditional entropy, and the second attribute is obtained by filtering based on the information gain of each attribute. Obtain a third attribute specified by the user; the plurality of attributes includes one or more of the first attribute, the second attribute, and the third attribute.

6. The method according to any one of claims 1 to 5, characterized in that, Before acquiring the object data of the object to be identified, the method further includes: Received object tracking voice; The target object is obtained by tracing the speech of the object; When it is confirmed that the target object is not the speaker of the object tracking voice, the search is performed within a predetermined space, and the searched object is identified as the object to be identified.

7. An object tracking device, characterized in that, Suitable for robots, the device includes: The acquisition module is used to: acquire object data of the object to be identified; The third recognition module is used to: when the object data includes a clear face image, perform face recognition based on the clear face image to determine whether the object to be recognized is the target object; when the object data does not include the clear face image or the face recognition fails, use the first recognition model or the second recognition model for identity recognition. The first identification module is used to: when the object data includes an object image and voiceprint information, use a first identification model to extract features from the object image and the voiceprint information and perform cross-modal feature fusion, so as to determine whether the object to be identified is the target object based on the cross-modal fusion features; The second identification module is used to: when the object data includes the object image, employ a second identification model to perform identity recognition based on the object image to determine whether the object to be identified is the target object; the process of employing the second identification model to perform identity recognition based on the object image to determine whether the object to be identified is the target object includes: performing multi-scale feature extraction on the object image to obtain a fifth feature; fusing deep and shallow features of the fifth feature and performing enhancement processing based on a multi-head attention mechanism to obtain a sixth feature; performing multi-attribute identity recognition based on the sixth feature to obtain recognition results for multiple attributes; and so on. The weights of the multiple attributes are generated based on the occlusion information in the object image, and the recognition results of the multiple attributes are weighted and summed to obtain a comprehensive recognition result; the object to be identified is determined as the target object based on the comprehensive recognition result; the step of multi-scale feature extraction of the object image to obtain a fifth feature includes: extracting and fusing features from multiple block images of the object image based on an attention mechanism to obtain shallow features; downsampling the shallow features and processing them based on an attention mechanism to obtain deep features; and obtaining the fifth feature based on the shallow features and the deep features.

8. A storage medium, characterized in that, It stores a computer program that, when executed by the robot's processor, causes the robot to perform the method described in any one of claims 1 to 6.

9. A robot, characterized in that, include: Memory, which stores computer programs; A processor reads a computer program stored in memory to perform the method described in any one of claims 1 to 6.

Images