Target sensing method and device, robot, storage medium and program product

Abstract

The invention provides a target sensing method and device, a robot, a storage medium and a program product, and relates to the technical field of computers. The target sensing method comprises the following steps: distributing tracking identifiers for targets included in a plurality of to-be-processed images; according to the sequence of the algorithm serial numbers from small to large, each target recognition algorithm in N target recognition algorithms is used for recognizing the target until the identity label of the target is recognized, N is a natural number larger than 1, and the recognition confidence coefficient of the target recognition algorithms is decreased along with the increase of the algorithm serial numbers; the target feature vectors used by the plurality of target recognition algorithms are different from one another. According to the invention, the identity information of the human body target can be accurately obtained under the conditions of single collected data and limited computing power.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a target perception method and device, a robot, a storage medium, and a program product. Background Art

[0002] The perception, positioning and tracking of human targets are classic problems in the field of computer vision. The human target perception system is used to achieve the perception, positioning and tracking of human targets based on vision, and has been applied in many scenarios including smart security, autonomous driving, smart parks, robots, etc. The existing human target perception system processes multiple video streams to obtain perception results. Summary of the invention

[0003] The inventors have noticed that in robot scenarios, identity recognition is much more important than in other scenarios. If the robot cannot recognize the identity information of a human target, or misidentifies the identity information of a human target, the subsequent human-computer interaction experience of the robot will be greatly reduced.

[0004] In the related art, the following characteristics exist in the robot scene.

[0005] 1) Single data: When the power consumption and cost of the robot are limited, the data collected by the robot is relatively single. Usually, the robot uses a single image acquisition device to achieve complete information perception of human targets, making the multi-sensor, multi-view fusion verification strategy commonly used in autonomous driving scenarios unusable in robot scenarios.

[0006] 2) Limited computing power: Under relatively low power consumption conditions, robots cannot provide computing power close to that of autonomous driving scenarios. Therefore, human target perception systems with high computing power requirements cannot be effectively deployed in robot scenarios.

[0007] Based on this, the present disclosure provides a target perception method that can accurately obtain the identity information of a human target when the collected data is single and the computing power is limited.

[0008] In a first aspect of the present disclosure, a target perception method is provided, comprising: assigning tracking identifications to targets included in a plurality of images to be processed; identifying the targets using each of N target recognition algorithms in ascending order of algorithm numbers until the identity of the target is identified, N being a natural number greater than 1, the recognition confidence of the target recognition algorithm decreasing as the algorithm number increases, and the target feature vectors used by the plurality of target recognition algorithms are different from each other.

[0009] In some embodiments, identifying the target includes: in the process of identifying the target using the i-th target recognition algorithm, identifying the i-th target feature vector of the target, ; Determine whether the i-th information library includes identity information that matches the i-th target feature vector; if the i-th information library includes the identity information, use the identity information as the identity of the target.

[0010] In some embodiments, the determining whether the i-th information library includes identity information that matches the i-th target feature vector includes: calculating the similarity between the i-th target feature vector and each candidate feature vector in the i-th information library to obtain multiple similarities; determining whether the maximum similarity among the multiple similarities is greater than a similarity threshold; when the maximum similarity is greater than the similarity threshold, using the candidate feature vector corresponding to the maximum similarity as a matching feature vector; and using the identity information corresponding to the matching feature vector as the identity information that matches the i-th target feature vector.

[0011] In some embodiments, the association between the tracking identifier and the identity identifier of the target is written into a cache.

[0012] In some embodiments, when i is 1, the first target feature vector is written into the first field corresponding to the identity information in the first information library.

[0013] In some embodiments, the total number of current feature vectors in the first field is detected; if the total number of current feature vectors is greater than a total threshold, each feature vector in the first field is deleted in order of writing time until the total number of feature vectors in the first field is no greater than the total threshold.

[0014] In some embodiments, obtaining the j-th target feature vector of the target, ; Write the jth target feature vector into the jth field corresponding to the identity information in the jth information library.

[0015] In some embodiments, the total number of current feature vectors in the j-th field is detected; if the total number of current feature vectors is greater than a total threshold, each feature vector in the j-th field is deleted in sequence according to the order of writing time until the total number of feature vectors in the j-th field is no greater than the total threshold.

[0016] In some embodiments, identifying the target includes: in the process of identifying the target using the Nth target recognition algorithm, identifying the Nth target feature vector of the target; calculating the similarity between the Nth target feature vector and the feature vector of each candidate identity information in the Nth information library to obtain multiple similarities; and using the candidate identity information corresponding to the maximum similarity among the multiple similarities as the identity of the target.

[0017] In some embodiments, when the target includes M sub-targets, it is detected whether there is an identical identity identifier among the identity identifiers of the M sub-targets, where M is a natural number greater than 1; if K sub-targets among the M sub-targets have the same identity identifier, then the algorithm serial number of the target recognition algorithm used to identify the identity identifier of each sub-target among the K sub-targets is determined to obtain K algorithm serial numbers, ; Among the K algorithm numbers, delete the identity identifiers corresponding to the K-1 algorithm numbers except the smallest algorithm number; if the association relationship between the identity identifier corresponding to the t-th algorithm number among the K-1 algorithm numbers and the tracking identifier has been written into the cache, delete the association relationship from the cache, .

[0018] In some embodiments, after assigning tracking identifiers to the targets included in the multiple images to be processed, it also includes: detecting whether the cache includes identity information corresponding to the tracking identifier; if the cache includes identity information corresponding to the tracking identifier, using the identity information as the identity of the target; if the cache does not include identity information corresponding to the tracking identifier, executing in ascending order of algorithm sequence numbers, using each target recognition algorithm in N target recognition algorithms to identify the target.

[0019] In some embodiments, each of the multiple images to be processed includes a plane image and a depth image collected simultaneously, and the plane image and the depth image both include the target; the step of assigning a tracking identifier to the target included in the multiple images to be processed includes: detecting the area information of the target in each plane image; detecting the plane position information of each key point of the target based on the area information; acquiring the depth information of each key point based on the depth image corresponding to each plane image; determining the three-dimensional spatial position information of each key point based on the plane position information of each key point and the depth information of each key point; tracking the target based on the three-dimensional spatial position information, and assigning the tracking identifier to the target based on the tracking result.

[0020] In some embodiments, the multiple images to be processed are filtered according to preset rules to obtain a target image; and the target image is sent to a server so that the server performs multimodal processing on the target image using a multimodal model.

[0021] In some embodiments, when N is 3, the first target feature vector used by the first target recognition algorithm is the facial feature vector of the target; the second target feature vector used by the second target recognition algorithm is the body feature vector of the target; and the third target feature vector used by the third target recognition algorithm is the body attribute feature vector of the target.

[0022] In some embodiments, the identity information of the target is recorded, wherein the recording of the identity information of the target includes: acquiring an image to be detected; in a case where the image to be detected includes the target, detecting whether a designated area of ​​the target satisfies a predetermined condition; in a case where the designated area of ​​the target satisfies the predetermined condition, acquiring a target feature vector of the designated area of ​​the target; writing the identity identifier assigned to the target and the first target feature vector of the designated area into a first information base; and writing the received Nth target feature vector of the target into an Nth information base.

[0023] In a second aspect of the present disclosure, a target perception device is provided, comprising: a memory; a processor, coupled to the memory, the processor being configured to execute a target perception method as described in any of the above embodiments based on instructions stored in the memory.

[0024] In a third aspect of the present disclosure, a robot is provided, comprising: a target perception device as described in any of the above embodiments; and an image acquisition device configured to acquire images and send the images to the target perception device.

[0025] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and when the instructions are executed by a processor, the target perception method as described in any of the above embodiments is implemented.

[0026] In a fifth aspect of the present disclosure, a computer program product is provided, comprising computer instructions, wherein the computer instructions, when executed by a processor, implement the target perception method as described in any of the above embodiments.

[0027] Other features and advantages of the present disclosure will become apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0029] Figure 1 The present invention is a flowchart of a target perception method according to an embodiment of the present invention.

[0030] Figure 2 A flowchart of a target perception method according to another embodiment of the present disclosure.

[0031] Figure 3 The present invention is a flowchart of a target perception method according to another embodiment of the present invention.

[0032] Figure 4 The present invention is a flowchart of a target perception method according to another embodiment of the present invention.

[0033] Figure 5 A schematic diagram of the structure of a target sensing device according to an embodiment of the present disclosure.

[0034] Figure 6 The figure is a schematic diagram of the structure of a robot according to an embodiment of the present invention.

[0035] Figure 7 The present invention is a flowchart of a target perception method according to another embodiment of the present invention.

[0036] Figure 8 The figure is a flowchart of an identity entry method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present disclosure and its application or use. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0038] Unless specifically stated otherwise, the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure.

[0039] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0040] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0041] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0042] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0043] Figure 1 The following is a flow chart of a target sensing method according to an embodiment of the present disclosure. In some embodiments, the following target sensing method is performed by a target sensing device in a robot, including steps 11-12.

[0044] In step 11, tracking identifiers are assigned to targets included in a plurality of images to be processed.

[0045] In some embodiments, each of the multiple images to be processed includes a planar image and a depth image collected simultaneously, and both the planar image and the depth image include a target.

[0046] For example, a depth camera may be used to collect a planar image and a depth image. The planar image may be an RGB image.

[0047] In some embodiments, the step of assigning tracking marks to targets included in a plurality of images to be processed includes the following steps S1-S5.

[0048] S1) Detect the target area information in each plane image.

[0049] For example, a human body detection model such as Yolov8 is used to process a plane image to detect a human body region in the plane image. The detection result is shown in the following formula (1).

[0050] (1).

[0051] In expression (1), is the pixel coordinate of the upper left corner of the human target detection box in the image, is the height of the detection box area, is the width of the detection box area, is the confidence score of the target.

[0052] S2) Detect the plane position information of each key point of the target based on the area information.

[0053] For example, a human key point detection model such as Yolov8 is used to detect the human body region to detect the position information of the human key points. If 17 human key points are detected, the detection result is shown in the following formula (2).

[0054] (2).

[0055] In expression (2), is the coordinate information of the first human key point, and so on.

[0056] S3) acquiring depth information of each key point according to the depth image corresponding to each plane image.

[0057] For example, the coordinates of each key point are used to obtain the depth information of the key point in the depth image, as shown in the following formula (3).

[0058] (3).

[0059] In expression (3), is the depth information of the first human key point, and so on.

[0060] S4) determining the three-dimensional spatial position information of each key point according to the plane position information of each key point and the depth information of each key point.

[0061] For example, the depth information is mapped to the three-dimensional space under the camera coordinate system to obtain the position of each key point in the three-dimensional space, as shown in the following formula (4).

[0062] (4).

[0063] In expression (4), is the 3D coordinate value of the first key point of the human body, and so on.

[0064] S5) Tracking the target according to the three-dimensional spatial position information, and assigning a tracking identifier to the target according to the tracking result.

[0065] For example, the same human target detected in multiple consecutive frames of images is given a unique identifier. express.

[0066] It should be noted here that in the home scenario, since there are fewer human targets, the human movement speed is low, and the trajectory complexity is low, in order to reduce the computing resource overhead as much as possible, the Kalman filter + Hungarian matching method can be used to track the target.

[0067] Here we select the pixel center of the target on the plane image and the aspect ratio of the target ,area and average depth As the prediction value of Kalman filtering, the calculation formula is shown in formula (5) to formula (9).

[0068] (5).

[0069] (6).

[0070] (7).

[0071] (8).

[0072] (9).

[0073] Applying the Kalman filter to a sequence of detection results of a continuous target, the predicted value of the detection result of the t+1th frame can be obtained based on the results of frames tn to t. The predicted value of the t+1 frame obtained based on the Kalman filter is matched with the true value obtained by the detection model of the t+1 frame using the Hungarian matching method. If the match is successful, a corresponding tracking identifier is assigned to the target represented by the true value of the t+1 frame. If the match fails, a new Kalman filter is created to start predicting and tracking the new sequence of detection results. This solution can ensure the closure effect of a small number of moving targets in limited scenarios of robot applications. Depth information The introduction of can effectively solve the problem of tracking loss or error in traditional detection box-based Kalman filter after target overlap and occlusion, achieving a balance between effect and computing resource overhead.

[0074] In some embodiments, multiple images to be processed are filtered according to preset rules to obtain a target image, and the target image is sent to a server so that the server performs multimodal processing on the target image using a multimodal model.

[0075] It should be noted here that high-level semantic processing of images requires a large amount of computing resources. Since the computing power of the robot is limited, the problem of insufficient computing power of the robot itself can be effectively solved by handing over complex tasks to servers deployed in the cloud and having the server call the multimodal model for processing.

[0076] For example, the multimodal model may be a VLM (Visual Language Model), whereby the VLM model is used to simultaneously process vision (images) and language (text) to perform high-order visual language tasks, such as recognizing emotions and behaviors of human targets.

[0077] In step 12, the target is identified using each of the N target recognition algorithms in the order of algorithm numbers from small to large until the identity of the target is identified, N is a natural number greater than 1, and the recognition confidence of the target recognition algorithm decreases as the algorithm number increases. The target feature vectors used by multiple target recognition algorithms are different.

[0078] For example, when N is 3, the first target feature vector used by the first target recognition algorithm is the target's facial feature vector, the second target feature vector used by the second target recognition algorithm is the target's body feature vector, and the third target feature vector used by the third target recognition algorithm is the target's body attribute feature vector.

[0079] It should be noted here that among the above three target recognition algorithms, since the confidence of recognition using facial feature vectors is the highest and the confidence of recognition using human attribute feature vectors is the lowest, the recognition confidence of the first target recognition algorithm is the highest and the recognition confidence of the third target recognition algorithm is the lowest.

[0080] It should be noted here that in the process of using each of the N target recognition algorithms to recognize the target, the first target recognition algorithm is used to recognize the target. If the first target recognition algorithm can recognize the identity of the target, then other target recognition algorithms are no longer used to recognize the target. On the contrary, if the first target recognition algorithm cannot recognize the identity of the target, then the second target recognition algorithm is used to recognize the target, and so on.

[0081] In some embodiments, Figure 2 As shown, in the process of identifying the target using the i-th target recognition algorithm among N target recognition algorithms, the following steps 21-23 are performed, wherein .

[0082] In step 21, the i-th target feature vector of the target is identified.

[0083] In step 22, it is determined whether the i-th information base includes identity information that matches the i-th target feature vector.

[0084] For example, the above step of determining whether the i-th information base includes identity information matching the i-th target feature vector includes the following steps 221-224.

[0085] 221) Calculate the similarity between the i-th target feature vector and each candidate feature vector in the i-th information base to obtain multiple similarities.

[0086] 222) Determine whether the maximum similarity among multiple similarities is greater than a similarity threshold.

[0087] 223) When the maximum similarity is greater than the similarity threshold, the candidate feature vector corresponding to the maximum similarity is used as the matching feature vector.

[0088] 224) The identity information corresponding to the matching feature vector is used as the identity information matched with the i-th target feature vector.

[0089] In step 23, if the i-th information base includes identity information, the identity information is used as the identity of the target.

[0090] For example, the i-th target feature vector is a face feature vector. The i-th information library is a face feature library, which includes multiple identity identification information and a face feature vector corresponding to each identity identification information.

[0091] For example, identity information The storage format of the corresponding multiple facial feature vectors is shown in the following formula (10).

[0092] (10).

[0093] If the i-th target feature vector of the target is , the eigenvector and the feature vector in the i-th information base The similarity of is the largest, and the similarity is greater than the similarity threshold, then the feature vector As the matching feature vector, since the feature vector and identification information Correspondingly, the identity information As the target's identity, that is, the target's identity is .

[0094] It should be noted here that if the eigenvector and the feature vector in the i-th information base The similarity is the largest, but the similarity is less than the similarity threshold, which indicates that the i-th target recognition algorithm currently used cannot recognize the identity of the target.

[0095] In some embodiments, the association relationship between the tracking identifier and the target's identity identifier is written into a cache so that the identity identifier associated with the tracking identifier can be directly queried through the cache.

[0096] For example, if the target's tracking ID is , the identified identity is , then the content stored in the cache is as shown in the following formula (11).

[0097] (11).

[0098] In expression (11), The source of the identity, To identify the identity using the current i-th target recognition algorithm When , the similarity between the i-th target feature vector of the target and the matching feature vector in the i-th information library.

[0099] In some embodiments, when i is 1, the first target feature vector is written into the first field corresponding to the identity information in the first information base.

[0100] It should be noted here that since the recognition confidence of the first target recognition algorithm is the highest, the identity identified by the first target recognition algorithm can be considered correct. At the same time, the first target feature vector extracted by the first target recognition algorithm can accurately describe the latest status of the target. Therefore, by writing the first target feature vector of the target into the first field corresponding to the identity information in the first information database, it is possible to record subtle changes in the feature vector of the target, so as to further improve the success rate of recognition.

[0101] For example, in the first information base, the identity information The corresponding fields are shown in formula (10). If the identity identified by the first target recognition algorithm is , then the target's current facial feature vector ( ) is written into the first information base, then the identity information The corresponding fields are shown in formula (12).

[0102] (12).

[0103] In some embodiments, after the first target feature vector of the target is written into the first field corresponding to the identity information in the first information base, the total number of current feature vectors in the first field is further detected. If the total number of current feature vectors is greater than the total number threshold, each feature vector in the first field is deleted in sequence according to the order of writing time until the total number of feature vectors in the first field is no greater than the total number threshold.

[0104] It should be noted here that, under normal circumstances, the target feature vector will change to a certain extent over time. Compared with the target feature vector written earlier, the target feature vector written currently is more conducive to the recognition of the target. Therefore, when the field corresponding to the identity information includes too many target feature vectors, deleting the target feature vector written earlier can effectively save storage space without affecting the recognition effect.

[0105] For example, in the first information base, the identity information The corresponding fields are shown in formula (12). By deleting the target feature vector written earlier , then with the identity information The corresponding fields are shown in formula (13).

[0106] (13).

[0107] In some embodiments, when the first target feature vector is written into the first field corresponding to the identity information in the first information base, the jth target feature vector of the target is further obtained. , write the jth target feature vector into the jth field corresponding to the identity information in the jth information base.

[0108] It should be noted here that, since the recognition confidence of the first target recognition algorithm is the highest, the identity identified by the first target recognition algorithm can be considered correct. In this case, other feature vectors of the target are provided and stored in the field corresponding to the identity information in the corresponding information library, so that the slight changes of other feature vectors of the target can be recorded, so as to further improve the success rate of recognition.

[0109] For example, after the target's facial feature vector is written into a first information library for storing facial feature vectors, the target's human body feature vector is further written into a second information library for storing human body feature vectors.

[0110] In some embodiments, after the jth target feature vector is written into the jth field corresponding to the identity information in the jth information library, the total number of current feature vectors in the jth field is further detected. If the total number of current feature vectors is greater than the total number threshold, each feature vector in the jth field is deleted in sequence according to the order of writing time until the total number of feature vectors in the jth field is no greater than the total number threshold.

[0111] Since the target feature vector will change to a certain extent over time, the currently written target feature vector is more conducive to target recognition than the target feature vector written earlier. Therefore, when the field corresponding to the identity information includes too many target feature vectors, deleting the target feature vector written earlier can effectively save storage space without affecting the recognition effect.

[0112] In some embodiments, Figure 3 As shown, in the process of identifying a target using the Nth target recognition algorithm among N target recognition algorithms, the following steps 31-33 are performed.

[0113] In step 31, the Nth target feature vector of the target is identified.

[0114] In step 32, the similarity between the Nth target feature vector and the feature vector of each candidate identity identification information in the Nth information base is calculated to obtain a plurality of similarities.

[0115] In step 33, the candidate identity information corresponding to the maximum similarity among the multiple similarities is used as the target identity.

[0116] It should be noted here that since the first N-1 target recognition algorithms among the N target recognition algorithms have not recognized the target's identity, the Nth target recognition algorithm is required to recognize the target's identity. Among the multiple similarities calculated above, the candidate identity information corresponding to the maximum similarity is used as the target's identity, and it is not determined whether the maximum similarity is greater than a predetermined threshold, resulting in a low confidence level in the recognition result. In this case, the association between the tracking identifier and the recognized identity is not written into the cache.

[0117] In some embodiments, when a target includes M sub-targets, M is a natural number greater than 1, and the following steps S21-S24 are performed.

[0118] S21) Detect whether there is an identical identity among the identities of the M sub-targets.

[0119] S22) If K sub-targets among the M sub-targets have the same identity, then determine the algorithm sequence number of the target recognition algorithm used to identify the identity of each sub-target among the K sub-targets to obtain K algorithm sequence numbers, .

[0120] S23) Among the K algorithm numbers, the identity identifiers corresponding to K-1 algorithm numbers except the smallest algorithm number are deleted.

[0121] S24) If the association relationship between the identity identifier and the tracking identifier corresponding to the tth algorithm serial number among the K-1 algorithm serial numbers has been written into the cache, then the association relationship is deleted from the cache. .

[0122] It should be noted here that if the image contains two targets, and the tracking mark of target 1 is , the tracking identifier of target 2 is , but the identities of target 1 and target 2 are , which indicates that an error occurred during the target identification process. To resolve this error, it is necessary to detect the identification methods of target 1 and target 2. For example, the identification of target 1 is identified using the first target recognition algorithm, and the identity of target 2 It is identified by the second target recognition algorithm. Since the recognition confidence of the first target recognition algorithm is greater than the recognition confidence of the second target recognition algorithm, the identity of target 2 can be determined. Identification error. In this case, remove the identity of target 2 This ensures that in one frame of image, multiple targets do not share the same identity.

[0123] In addition, if the tracking marker of target 2 and identity The association relationship has been written into the cache. To avoid the incorrect association relationship from causing erroneous effects on target identification, delete the tracking identifier from the cache. and identity The associated relationship.

[0124] In the target perception method provided by the above embodiment of the present disclosure, multiple target recognition algorithms with different recognition confidences are set, and each of the multiple target recognition algorithms is used to recognize the target in descending order of recognition confidence until the identity of the target is recognized. In this way, the identity of the target can be accurately recognized using less data under limited computing power.

[0125] In some embodiments, the target registration is achieved by recording the target's identity information. For example, the step of recording the target's identity information includes steps S31-S35.

[0126] S31) Acquire an image to be detected.

[0127] S32) When the image to be detected includes a target, detecting whether a designated area of ​​the target satisfies a predetermined condition.

[0128] For example, the predetermined conditions include: whether the confidence of face detection is greater than a threshold, whether the current image includes only one face, etc.

[0129] S33) When the designated area of ​​the target satisfies a predetermined condition, a target feature vector of the designated area of ​​the target is obtained.

[0130] S34) writing the identity assigned to the target and the first target feature vector of the designated area into the first information base.

[0131] S35) writing the identity assigned to the target and the Nth target feature vector of the target into the Nth information base.

[0132] It should be noted here that among the N target recognition algorithms, the first target recognition algorithm has the highest recognition confidence. In order to ensure the recognition effect of the first target recognition algorithm, when registering the target, the first target feature vector of the target used by the first target recognition algorithm is written into the first information library corresponding to the first target recognition algorithm.

[0133] In addition, among the N target recognition algorithms, the recognition confidence of the Nth target recognition algorithm is the lowest. In order to ensure the recognition effect of the Nth target recognition algorithm, when registering the target, the Nth target feature vector of the target used by the Nth target recognition algorithm is written into the Nth information library corresponding to the Nth target recognition algorithm.

[0134] Figure 4 4 is a flow chart of a target sensing method according to another embodiment of the present disclosure. In some embodiments, the following target sensing method is performed by a target sensing device in a robot, including steps 41-44.

[0135] In step 41, tracking identifiers are assigned to targets included in a plurality of images to be processed.

[0136] In step 42, it is detected whether the cache includes identity information corresponding to the tracking identifier.

[0137] If the cache includes the identity information corresponding to the tracking identifier, then execute step 43. Otherwise, execute step 44.

[0138] In step 43, the identity information corresponding to the tracking identifier in the cache is used as the identity of the target.

[0139] In step 44, each of the N target recognition algorithms is used to recognize the target in the ascending order of the algorithm number.

[0140] It should be noted here that if the cache stores the association between the tracking identifier and the identity identifier, the identity identifier corresponding to the tracking identifier can be directly obtained through the cache. In this case, there is no need to use the target recognition algorithm for target recognition, which can effectively save computing resources.

[0141] Figure 5 A schematic diagram of the structure of a target sensing device according to an embodiment of the present disclosure.

[0142] like Figure 5 As shown, the target sensing device 50 can be in the form of a general-purpose computing device. The target sensing device 50 includes a memory 51, a processor 52, and a bus 53 connecting different system components.

[0143] The memory 51 may include, for example, a system memory, a non-volatile storage medium, etc. The system memory may store, for example, an operating system, an application program, a boot loader, and other programs. The system memory may include a volatile storage medium, such as a random access memory (RAM) and / or a cache memory. The non-volatile storage medium may store, for example, instructions of a corresponding embodiment of at least one target perception method being executed. The non-volatile storage medium includes, but is not limited to, a disk memory, an optical memory, a flash memory, and the like.

[0144] The processor 52 may be implemented by a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistors, and other discrete hardware components. Accordingly, each module such as an acquisition module, a calculation module, and an adjustment module may be implemented by a central processing unit (CPU) running instructions in a memory that execute corresponding steps, or may be implemented by a dedicated circuit that executes corresponding steps.

[0145] For example, the processor 52 is configured to execute instructions stored in the memory to implement the following Figures 1 to 4 The method of any one of the embodiments.

[0146] The bus 53 may use any of a variety of bus structures, including, but not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, and a Peripheral Component Interconnect (PCI) bus.

[0147] The interfaces 54, 55, 56 of the target sensing device 50, the memory 51 and the processor 52 can be connected via a bus 53. The input / output interface 54 can provide a connection interface for input / output devices such as a display, a mouse, and a keyboard. The network interface 55 provides a connection interface for various networked devices. The storage interface 56 provides a connection interface for external storage devices such as a floppy disk, a USB flash drive, and an SD card.

[0148] Here, various aspects of the present disclosure are described with reference to flowcharts and / or block diagrams of methods, devices, and computer program products according to embodiments of the present disclosure. It should be understood that each frame of the flowchart and / or block diagram and the combination of frames can be implemented by computer-readable program instructions.

[0149] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable device to produce a machine, so that the processor executes the instructions to produce means for implementing the functions specified in one or more blocks in the flowchart and / or block diagram.

[0150] These computer-readable program instructions may also be stored in a computer-readable memory, which cause the computer to work in a specific manner to produce an article of manufacture, including instructions for implementing the functions specified in one or more blocks in the flowchart and / or block diagram.

[0151] The present disclosure can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects.

[0152] The present disclosure also provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and when the instructions are executed by a processor, the following is implemented: Figures 1 to 4 The method of any one of the embodiments.

[0153] The present disclosure also provides a computer program product, including computer instructions, wherein when the computer instructions are executed by a processor, the following is achieved: Figures 1 to 4 The method of any one of the embodiments.

[0154] Figure 6 The figure is a schematic diagram of the structure of a robot according to an embodiment of the present invention.

[0155] like Figure 6 As shown, the robot 60 includes a target sensing device 50 and an image acquisition device 61. The target sensing device 50 is Figure 5 The target sensing device involved in any embodiment.

[0156] The image acquisition device 61 is configured to acquire images and send the images to the target perception device 50 .

[0157] In some embodiments, the image acquisition device 61 is a depth camera that can simultaneously acquire a planar image and a depth image.

[0158] The present disclosure is described below by using a specific example.

[0159] For example, the target perception method uses three target recognition algorithms, wherein the first target recognition algorithm uses a face feature vector, the second target recognition algorithm uses a body feature vector, and the third target recognition algorithm uses a body attribute feature vector.

[0160] It should be noted here that among the above three target recognition algorithms, since the confidence of recognition using facial feature vectors is the highest and the confidence of recognition using human attribute feature vectors is the lowest, the recognition confidence of the first target recognition algorithm is the highest and the recognition confidence of the third target recognition algorithm is the lowest.

[0161] like Figure 7 As shown, first, through the identity entry process, the target's facial features are written into the facial feature library, and the target's body attribute features are written into the body attribute library. Next, the target is identified using the cache, the first target recognition algorithm, the second target recognition algorithm, and the third target recognition algorithm in sequence until the target's identity is identified.

[0162] like Figure 8 As shown, the identity entry process includes steps 81-86.

[0163] In step 81, a human body detection model is used to perform human body detection on the collected image.

[0164] In step 82, based on the human body detection result, it is determined whether there is a human body in the image.

[0165] If there is a human body in the image, then execute step 83. Otherwise, execute step 86.

[0166] In step 83, the face detection model is used to detect the face.

[0167] In step 84, it is determined whether the face detection result is legal.

[0168] For example, the following conditions may be used to determine whether the face detection result is legal.

[0169] 1) Whether the confidence of face detection is greater than the threshold. This value can be used to ensure that the current user is facing the camera.

[0170] 2) Whether there is one and only one face in the current image, and whether the pixel area of ​​the face region is greater than the set value.

[0171] 3) Whether there is a depth difference when the current facial key points are mapped to the depth map is used to determine whether it is a real face and prevent the use of photos or face images displayed on the screen for registration.

[0172] If the face detection result is legal, execute step 85, otherwise execute step 86.

[0173] In step 85, the face detection result is processed using the face feature model to obtain a face feature vector. , and the facial feature vector and the identity assigned to the human body The corresponding relationship is stored in the face feature database.

[0174] In addition, the human body attribute vector corresponding to the human body attribute information of the human body is obtained. , the human attribute vector and the person's identity The corresponding relationship is stored in the human body attribute library.

[0175] It should be noted that in different application scenarios, the attributes of the human body can be set differently. For example, in a home application scenario, the human body attribute information may include the following attributes: (1) whether the person is a woman, (2) whether the person is a child, (3) whether the person is wearing glasses, (4) whether the person is wearing a hat, and (5) whether the person is obese. It includes 5 information bits, each of which is represented by 0 or 1. If the information bit is 1, it means that the corresponding attribute exists. For example, the human body attribute vector If the first information bit is 1, it indicates that the target person is a female, and if the second information bit is 0, it indicates that the target person is not a child.

[0176] In step 86, an error message is returned.

[0177] return Figure 7 ,When executing the target perception process, the following steps are performed.

[0178] 1) Use a lightweight human body detection model to process the collected planar image to detect the human body area in the planar image. The detection result is shown in the above formula (1).

[0179] For example, human detection models include Yolov8.

[0180] 2) Use a lightweight human key point detection model to process the human body area and detect the planar position information of each key point of the target human body.

[0181] For example, a human body key point detection model such as Yolov8 is used to detect the human body region and 17 human body key points are detected. The detection result is shown in the above formula (2).

[0182] 3) According to the depth image corresponding to each plane image, the depth information of each key point is obtained. According to the plane position information of each key point and the depth information of each key point, the three-dimensional spatial position information of each key point is determined.

[0183] For example, if 17 key points of the target human body are detected, the depth information of these 17 key points is as shown in the above formula (3). The depth information is mapped to the three-dimensional space under the camera coordinate system to obtain the position of each key point in the three-dimensional space, as shown in the above formula (4).

[0184] Next, step 4.1 and step 4.2 may be performed in parallel.

[0185] 4.1) According to preset rules, multiple images to be processed are filtered to obtain a target image, and the target image is sent to the server so that the server can use VLM to perform multimodal processing on the target image to identify the target person's emotions, behaviors, etc.

[0186] It should be noted here that high-level semantic processing of images requires a large amount of computing resources. Since the computing power of the robot is limited, the problem of insufficient computing power of the robot itself can be effectively solved by handing over complex tasks to servers deployed in the cloud and having the server call the multimodal model for processing.

[0187] 4.2) Track the target according to the three-dimensional spatial position information and assign a tracking identifier to the target based on the tracking result.

[0188] For example, the same human target detected in multiple consecutive frames of images is given a unique identifier. express.

[0189] It should be noted here that in the home scenario, since there are fewer human targets, the human movement speed is low, and the trajectory complexity is low, in order to reduce the computing resource overhead as much as possible, the Kalman filter + Hungarian matching method can be used to track the target.

[0190] 5) Check whether the cache contains the tracking identifier The corresponding identity information. If the cache contains the tracking identifier The corresponding identity information is used as the identity information of the target person. .

[0191] It should be noted here that if the cache stores the association between the tracking identifier and the identity identifier, the identity identifier corresponding to the tracking identifier can be directly obtained through the cache. In this case, there is no need to use the target recognition algorithm for target recognition, which can effectively save computing resources.

[0192] If the cache does not contain the tracking identifier The corresponding identity information may be in the following three situations: (1) This is a new target person; (2) The target person has been identified before, but due to occlusion or excessive movement or other reasons, the tracking identification is lost. Lost; (3) With the tracking mark The target person has not been identified effectively before. In this case, the identity of the target person needs to be re-identified.

[0193] 6) Obtain the face area of ​​the target person by using the face detection model.

[0194] For example, by using the insightface library, face area detection is achieved.

[0195] 7) Obtain the facial feature vector of the target person by using the face recognition model Next, calculate the facial feature vector The feature similarity with each face feature vector in the face feature library.

[0196] It should be noted here that in the face feature database, identity information The storage format of the corresponding multiple facial feature vectors is shown in the above formula (10).

[0197] If the target person's facial feature vector is , the eigenvector and the feature vector in the face feature library The similarity of is the largest, and the similarity is greater than the similarity threshold, then the feature vector as the matching feature vector. Since the feature vector and identification information Correspondingly, the identity information As the identity of the target person, that is, the identity of the target person is .

[0198] Since the identity of the target person has been identified, the subsequent target perception process is no longer executed.

[0199] It should be noted here that if the eigenvector and the feature vector in the face feature library The similarity is the largest, but the similarity is less than the similarity threshold, which means that the identity of the target face cannot be recognized by using the face feature vector.

[0200] After identifying the target person’s identity After that, you need to further execute the following process.

[0201] 7.1) Tracking mark of the target person and the identity is The association relationship is written into the cache, as shown in the above formula (11).

[0202] 7.2) The facial feature vector of the target person Write into the face feature library and identity In the corresponding field, as shown in the above formula (12), the face feature vector In the above formula (12), it is represented as the feature vector .

[0203] Next, detection and identification The total number of current face feature vectors in the corresponding field. If the total number of current feature vectors is greater than the total threshold, each feature vector in the field is deleted in the order of writing time until the total number of feature vectors in the field is no greater than the total threshold.

[0204] It should be noted here that, under normal circumstances, the facial feature vector of the target person will change to a certain extent over time. Compared with the facial feature vector written earlier, the facial feature vector written currently is more conducive to the recognition of the target person. Therefore, when too many facial feature vectors are included in this field, deleting the facial feature vector written earlier can effectively save storage space without affecting the recognition effect.

[0205] For example, in the above formula (12), by deleting the target feature vector written earlier , then with the identity The corresponding fields are shown in formula (13) above.

[0206] 7.3) Obtain the human feature vector of the target human body by using the human feature model , and the human feature vector Write the identity tag in the human feature database corresponding fields.

[0207] For example, the human feature model includes the Resnet18 model.

[0208] In the human body feature vector Write the identity tag in the human feature database After the corresponding field is entered, the total number of current human feature vectors in the field is further detected. If the total number of current human feature vectors is greater than the total threshold, each feature vector in the field is deleted in sequence according to the order of writing time until the total number of feature vectors in the field is no greater than the total threshold.

[0209] Since the human feature vector of the target human body will change to a certain extent over time, the currently written human feature vector is more conducive to the recognition of the target human body than the human feature vector written earlier. Therefore, when too many human feature vectors are included in this field, deleting the human feature vector written earlier can effectively save storage space without affecting the recognition effect.

[0210] 8) If the target person’s identity cannot be identified through the facial feature vector, use the human feature model to identify the target person’s human feature vector Next, calculate the human feature vector The feature similarity with each human feature vector in the human feature library.

[0211] It should be noted here that in the human feature database, identity information The storage format of the corresponding multiple human feature vectors is shown in the following formula (14).

[0212] (14).

[0213] If the target human body feature vector is , human feature vector and the feature vector in the human feature library The similarity of is the largest, and the similarity is greater than the similarity threshold, then the feature vector as the matching feature vector. Since the feature vector and identification information Correspondingly, the identity information As the identity of the target person, that is, the identity of the target person is .

[0214] It should be noted here that if the target person has his back to the robot, or the target person's face is not facing the robot, in this case, the robot cannot effectively obtain the facial feature information of the target person, but can obtain the body feature information of the target person. Therefore, if the identity of the target person cannot be identified through facial recognition, the identity of the target person can be effectively identified through body feature recognition.

[0215] Since the identity of the target person has been identified, the subsequent target perception process is no longer executed.

[0216] It should be noted here that if the eigenvector and the feature vector in the human feature library The similarity is the largest, but the similarity is less than the similarity threshold, which means that the identity of the target face cannot be recognized by using the human feature vector.

[0217] After identifying the target person’s identity After that, the target human body is tracked and the identity is The association relationship is written into the cache, and the storage format is shown in the following formula (15).

[0218] (15).

[0219] In expression (15), The source of the identity, To identify the identity When , the similarity between the human feature vector of the target human body and the matching feature vector in the human feature library is calculated.

[0220] It should also be noted that compared with face recognition, the recognition method based on human features is easily interfered. For example, if two people wear similar clothes or walk in a similar manner, it is easy to make a misjudgment. Therefore, after successfully identifying the identity of the target person based on human features, there is no need to update the face feature library and the human feature library.

[0221] 9) If the target person’s identity cannot be identified through the human feature vector, use the human attribute model to identify the target person’s human attribute feature vector Next, calculate the human attribute feature vector The feature similarity with each human attribute feature vector in the human attribute library.

[0222] It should be noted here that in the human attribute library, identity information The storage format of the corresponding human attribute feature vector is shown in formula (16).

[0223] (16).

[0224] If the target person's human attribute feature vector is , human attribute feature vector The attribute feature vector in the human attribute library The similarity is the largest, then the attribute feature vector Corresponding identity information As the identity of the target person, that is, the identity of the target person is .

[0225] It should also be noted that human attribute recognition is mainly used to determine some preset attributes of the current target human body, such as gender, age, accessories, body shape, etc. The attribute information of each target human body is written into the human attribute library during the identity entry process. Fuzzy matching is performed with each human attribute feature vector in the human attribute library. For example, a human attribute feature vector includes 4 information bits, where the first information bit corresponds to gender, the second information bit corresponds to age, the third information bit corresponds to accessories, and the fourth information bit corresponds to body shape. If the human attribute feature vector of the target person The attribute feature vector in the human attribute library Compared with the above, if there are 3 information bits that are consistent, then the human attribute feature vector can be identified. The attribute feature vector in the human attribute library The similarity is 3.

[0226] It should be noted here that since the confidence of using human attribute feature vectors for identity recognition is low, the target human body will not be tracked. And the identity identified by the human attribute feature vector is The association relationship is written into the cache.

[0227] from Figure 8 It can be seen that the identity of the target person can be obtained through one of the four processing processes of caching, face recognition, human feature recognition and human attribute recognition. Therefore, through the identity aggregation logic, while effectively identifying the identity of the target person, it can also ensure that a target person has only one identity.

[0228] 10) If the image includes multiple target human bodies, and the multiple target human bodies have the same identity, then among these same identity identifiers, only the one with the highest confidence is retained, and the other identity identifiers are deleted. Among the above three identification methods of identifying identity through facial features, identifying identity through body features, and identifying identity through body attribute features, the identity identified through facial features has the highest confidence, and the identity identified through body attribute features has the lowest confidence.

[0229] If the image contains two target people, the tracking mark of target person 1 is , the tracking mark of target person 2 is , but the identity labels of target person 1 and target person 2 are . Identity of target person 1 It is identified by facial features, and the identity of the target person 2 It is identified by human features. Since the identity identified by facial features has the highest confidence, the identity of target person 2 is deleted. This ensures that in one frame of image, multiple targets do not share the same identity.

[0230] In addition, due to the tracking mark of the target human body 2 and identity The association relationship has been written into the cache. To avoid the incorrect association relationship from causing erroneous effects on target identification, delete the tracking identifier from the cache. and identity The associated relationship.

[0231] For another example, if the image includes two target human bodies, the tracking mark of target human body 1 is , the tracking mark of target person 2 is , but the identity labels of target person 1 and target person 2 are . Identity of target person 1 It is identified by facial features, and the identity of the target person 2 It is identified by human attributes. Since the identity identified by facial features has the highest confidence, the identity of target person 2 is deleted. This ensures that in one frame of image, multiple targets do not share the same identity.

[0232] In addition, due to the tracking mark of the target human body 2 and identity The association is not written to the cache, so there is no need to delete the tracking identifier from the cache. and identity The operation of the association relationship.

[0233] By implementing the above-mentioned embodiments of the present disclosure, the following beneficial effects can be obtained.

[0234] (1) By using multiple models to identify the identity of the target from different dimensions, the effectiveness of identity recognition can be guaranteed as much as possible.

[0235] (2) Construct an online feature update mechanism so that the stored features can represent different situations of the same target at different times.

[0236] (3) Through identity aggregation logic, while effectively identifying the identity of the target person, it can also ensure that a target person has only one identity.

[0237] (4) Run high-frequency, low-computing algorithms under local limited computing conditions, and run low-frequency, high-computing algorithms in the cloud. This ensures that basic human target perception capabilities can be provided when the robot's computing resources are limited, and that higher-level human semantic understanding can be provided with the help of cloud computing power.

[0238] In some embodiments, the functional unit described above may be implemented as a general-purpose processor, a programmable logic controller (PLC), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or any appropriate combination thereof, for performing the functions described in the present disclosure.

[0239] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0240] The description of the present disclosure is given for the purpose of illustration and description, and is not intended to be exhaustive or to limit the present disclosure to the disclosed form. Many modifications and variations will be apparent to those of ordinary skill in the art. The embodiments are selected and described in order to better illustrate the principles and practical applications of the present disclosure, and to enable those of ordinary skill in the art to understand the present disclosure and thereby design various embodiments with various modifications suitable for specific uses.

Claims

1. A target perception method, characterized in that: include: Assigning tracking identifiers to targets included in a plurality of images to be processed; In the order of algorithm numbers from small to large, each target recognition algorithm in N target recognition algorithms is used to recognize the target until the identity of the target is recognized, N is a natural number greater than 1, and the recognition confidence of the target recognition algorithm decreases as the algorithm number increases, and the target feature vectors used by the multiple target recognition algorithms are different.

2. The target perception method according to claim 1, characterized in that: The identifying of the target comprises: In the process of identifying the target using the i-th target recognition algorithm, identifying the i-th target feature vector of the target, ; Determining whether the i-th information base includes identity information matching the i-th target feature vector; If the identity information is included in the i-th information base, the identity information is used as the identity of the target.

3. The target perception method according to claim 2, characterized in that: The determining whether the i-th information base includes identity information matching the i-th target feature vector comprises: Calculating the similarity between the i-th target feature vector and each candidate feature vector in the i-th information base to obtain multiple similarities; Determine whether the maximum similarity among the multiple similarities is greater than a similarity threshold; When the maximum similarity is greater than the similarity threshold, taking the candidate feature vector corresponding to the maximum similarity as the matching feature vector; The identity identification information corresponding to the matching feature vector is used as the identity identification information matched with the i-th target feature vector.

4. The target perception method according to claim 2, characterized in that: Also includes: The association relationship between the tracking identifier and the identity identifier of the target is written into the cache.

5. The target perception method according to claim 2, characterized in that: Also includes: When i is 1, the first target feature vector is written into the first field corresponding to the identity information in the first information base.

6. The target perception method according to claim 5, characterized in that: Also includes: Detecting the total number of current feature vectors in the first field; If the total number of the current feature vectors is greater than the total number threshold, each feature vector in the first field is deleted in sequence according to the order of writing time until the total number of feature vectors in the first field is no greater than the total number threshold.

7. The target perception method according to claim 5, characterized in that: Also includes: Get the j-th target feature vector of the target, ; The j-th target feature vector is written into the j-th field corresponding to the identity identification information in the j-th information base.

8. The target perception method according to claim 7, characterized in that: Also includes: Detecting the total number of current feature vectors in the j-th field; If the total number of the current feature vectors is greater than the total number threshold, each feature vector in the j-th field is deleted in sequence according to the order of writing time until the total number of feature vectors in the j-th field is no greater than the total number threshold.

9. The target perception method according to claim 1, characterized in that: The identifying of the target comprises: In the process of identifying the target using the Nth target recognition algorithm, identifying the Nth target feature vector of the target; Calculating the similarity between the Nth target feature vector and the feature vector of each candidate identity identification information in the Nth information base to obtain multiple similarities; The candidate identity information corresponding to the maximum similarity among the multiple similarities is used as the identity of the target.

10. The target perception method according to claim 1, characterized in that: Also includes: In the case where the target includes M sub-targets, detecting whether there is an identical identity identifier among the identity identifiers of the M sub-targets, where M is a natural number greater than 1; If K sub-targets among the M sub-targets have the same identity, then determine the algorithm sequence number of the target recognition algorithm used to identify the identity of each sub-target among the K sub-targets to obtain K algorithm sequence numbers, ; Among the K algorithm numbers, the identity identifiers corresponding to K-1 algorithm numbers except the smallest algorithm number are deleted; If the association relationship between the identity identifier corresponding to the tth algorithm serial number among the K-1 algorithm serial numbers and the tracking identifier has been written into the cache, then deleting the association relationship from the cache, .

11. The target perception method according to claim 1, characterized in that: After assigning tracking identifiers to the targets included in the plurality of images to be processed, the method further includes: Detecting whether the cache includes identity information corresponding to the tracking identifier; If the cache includes identity information corresponding to the tracking identifier, using the identity information as the identity of the target; If the cache does not include the identity information corresponding to the tracking identifier, the target is identified by using each target identification algorithm in N target identification algorithms in the order of algorithm numbers from small to large.

12. The target perception method according to claim 1, characterized in that: Each of the plurality of images to be processed includes a planar image and a depth image collected simultaneously, and both the planar image and the depth image include the target; The step of assigning tracking identifiers to targets included in the plurality of images to be processed comprises: Detecting the area information of the target in each plane image; Detecting the plane position information of each key point of the target according to the area information; Acquire depth information of each key point according to the depth image corresponding to each planar image; Determine the three-dimensional spatial position information of each key point according to the plane position information of each key point and the depth information of each key point; The target is tracked according to the three-dimensional space position information, and the tracking identifier is assigned to the target according to the tracking result.

13. The target perception method according to claim 12, characterized in that: Also includes: According to a preset rule, the plurality of images to be processed are filtered to obtain a target image; The target image is sent to a server so that the server performs multimodal processing on the target image using a multimodal model.

14. The target perception method according to claim 1, characterized in that: When N is 3, The first target feature vector used by the first target recognition algorithm is the face feature vector of the target; The second target feature vector used by the second target recognition algorithm is the human feature vector of the target; The third target feature vector used by the third target recognition algorithm is the human attribute feature vector of the target.

15. The target perception method according to any one of claims 1 to 14, characterized in that: Also includes: Recording the identity information of the target, wherein the recording the identity information of the target includes: Acquire the image to be detected; In the case where the target is included in the image to be detected, detecting whether a designated area of ​​the target meets a predetermined condition; When the designated area of ​​the target satisfies the predetermined condition, obtaining a target feature vector of the designated area of ​​the target; Writing the identity assigned to the target and the first target feature vector of the designated area into a first information base; The identity assigned to the target and the Nth target feature vector of the target are written into the Nth information base.

16. A target sensing device, characterized in that: include: Memory; A processor is coupled to a memory, and the processor is configured to execute the target perception method as described in any one of claims 1-15 based on instructions stored in the memory.

17. A robot, characterized in that: include: The target sensing device as claimed in claim 16; The image acquisition device is configured to acquire images and send the images to the target perception device.

18. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the instructions are executed by a processor, the target perception method as described in any one of claims 1-15 is implemented.

19. A computer program product, characterized in that It comprises computer instructions, wherein when the computer instructions are executed by a processor, the target perception method as described in any one of claims 1-15 is implemented.

Images