Method, apparatus, device, storage medium and program product for robot following

Abstract

The present application discloses a method, device, equipment, storage medium and program product for robot following, which is applied to intelligent robots. The specific technical solutions include: obtaining an image to be recognized, a tracking linked list corresponding to a following target, and ultra-wideband UWB positioning data. The tracking linked list includes the predicted appearance features and predicted pose information of the following target; for the image content of each person, obtaining the appearance features and pose information of the person in the image content; calculating a fusion evaluation result corresponding to each person according to the appearance features and pose information of the person in the image content, the predicted appearance features, the predicted pose information and the UWB positioning data. In the case where the maximum fusion evaluation result is greater than a preset threshold value, taking the person corresponding to the maximum fusion evaluation result as the following target; following the following target according to the positioning information of the following target. Determining the following target from multiple dimensions improves the accuracy of the recognition result, thereby realizing accurate following of the robot.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, and particularly relates to a method, device, equipment, storage medium and program product for robot following. Background Art

[0002] Currently, a robot can obtain the position information of a following target through Ultra Wideband (UWB) technology, and obtain the image information of the following target through a vision sensor, and then fuse the position information and the image information to determine the position of the following target, so as to reasonably plan a following path and complete a following task.

[0003] However, in a scenario with multiple targets, the robot cannot accurately identify a specific following target, and thus cannot achieve accurate following. Summary of the Invention

[0004] Embodiments of this application provide a method, device, equipment, storage medium and program product for robot following. It can effectively identify a following target in a scenario with multiple people, so as to achieve accurate following of the robot.

[0005] In a first aspect, an embodiment of this application provides a method for robot following, which is applied to a robot. The method includes:

[0006] Obtain an image to be recognized, a tracking linked list corresponding to a following target, and the Ultra Wideband (UWB) positioning data of the following target. The tracking linked list includes the predicted appearance features and predicted pose information of the following target;

[0007] For the image content of each person in the image to be recognized, obtain the appearance features and pose information of the person in the image content;

[0008] According to the appearance features and pose information of the person in the image content, the predicted appearance features, predicted pose information of a preset following target, and the UWB positioning data, calculate a fusion evaluation result of the following target and each person in the image content. The fusion evaluation result is used to measure the probability that the person in the image to be recognized is the following target;

[0009] In the case where the maximum fusion evaluation result is greater than a preset threshold value, use the person corresponding to the maximum fusion evaluation result as the following target;

[0010] Follow the following target according to the positioning information of the following target.

[0011] In a possible implementation manner, the obtaining the appearance features and pose information of the person in the image content includes:

[0012] Input the image content into the re-identification model, and use the re-identification model to extract features from the image content to obtain the appearance features;

[0013] Input the image content into the detection model to obtain the pose information.

[0014] In a possible implementation manner, before obtaining the image to be recognized, the tracking linked list corresponding to the following target, and the ultra-wideband UWB positioning data of the following target, the method further includes:

[0015] Obtain a target image including the following target;

[0016] Input the target image into the re-identification model to obtain the predicted appearance features;

[0017] Input the target image into the detection model to obtain the predicted pose information;

[0018] Construct a tracking linked list including the corresponding relationship between the predicted appearance features and the predicted pose information.

[0019] In a possible implementation manner, calculating the fusion evaluation result of the following target and each person in the image content according to the appearance features and pose information of the person in the image content, as well as the predicted appearance features, predicted pose information of the preset following target, and the UWB positioning data, includes:

[0020] Calculate the similarity evaluation result between the predicted appearance features and the appearance features of each person;

[0021] Determine the predicted detection frame position information of the following target according to the predicted pose information, and determine the detection frame position information of each person according to the pose information of each person;

[0022] Based on the predicted detection frame position information and the detection frame position information of each person, calculate the intersection-over-union evaluation result between the predicted detection frame of the following target and the detection frame of each person;

[0023] Determine the predicted center point position corresponding to the following target based on the UWB positioning data, and determine the center point position corresponding to each person based on the detection frame position information of each person;

[0024] Calculate the position deviation evaluation result based on the predicted center point position and the center point position of each person;

[0025] Perform a weighted sum on the similarity evaluation result, the intersection-over-union evaluation result, and the position deviation evaluation result to obtain the fusion evaluation result.

[0026] In a possible implementation, before following the following target according to the positioning information of the following target, the method further includes:

[0027] Obtain the point cloud distance information of the point cloud corresponding to the following target;

[0028] Obtain the target point cloud information corresponding to the target point cloud according to the point cloud distance information and a preset distance ratio;

[0029] Calculate the distance mean value of the target point cloud based on the target point cloud information to obtain the detection distance of the following target;

[0030] Perform Kalman filtering on the UWB positioning data and the detection distance to obtain the positioning information.

[0031] In a possible implementation, after following the following target according to the positioning information of the following target, the method further includes:

[0032] In the case of determining that the following of the following target fails, receive audio data containing preset wake-up word information;

[0033] Calculate the position information of the following target according to the audio data;

[0034] Follow the following target according to the position information.

[0035] In a second aspect, an embodiment of the present application provides a robot following device, which is applied to an intelligent robot. The device includes:

[0036] An acquisition module, configured to acquire an image to be recognized, a tracking linked list corresponding to a following target, and ultra-wideband UWB positioning data of the following target. The tracking linked list includes the predicted appearance features and predicted pose information of the following target;

[0037] The acquisition module is further configured to acquire the appearance features and pose information of a person in the image content for each person in the image to be recognized;

[0038] A calculation module, configured to calculate a fusion evaluation result of the following target and each person in the image content according to the appearance features and pose information of the person in the image content, the predicted appearance features, predicted pose information of a preset following target, and the UWB positioning data. The fusion evaluation result is used to measure the probability that the person in the image to be recognized is the following target;

[0039] A determination module, configured to use the person corresponding to the maximum fusion evaluation result as the following target when the maximum fusion evaluation result is greater than a preset threshold;

[0040] A following module, configured to follow the following target according to the positioning information of the following target.

[0041] In a third aspect, an embodiment of the present application provides an electronic device, including: a processor and a memory storing computer program instructions;

[0042] When the processor executes the computer program instructions, the method for robot following as described in any item of the first aspect is implemented.

[0043] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the method for robot following as described in any item of the first aspect is implemented.

[0044] In a fifth aspect, an embodiment of the present application provides a computer program product, and when the instructions in the computer program product are executed by a processor of an electronic device, the electronic device is enabled to execute the method for robot following as described in any item of the first aspect.

[0045] For a method, device, equipment, storage medium, and program product for robot following according to an embodiment of the present application, the robot obtains a tracking linked list corresponding to the following target in the to-be-recognized image and ultra-wideband (UWB) positioning data of the following target. The tracking linked list includes predicted appearance features and predicted pose information of the following target, and appearance features and pose information corresponding to each person in the to-be-recognized image are obtained. Based on the predicted appearance features and predicted pose information of the following target, a fusion evaluation result of the predicted appearance features and predicted pose information and the appearance features and pose information of the person in the to-be-recognized image can be calculated, so as to determine the following target in the to-be-recognized image. In this way, the following target in the to-be-recognized image is judged from multiple dimensions based on appearance features, pose information, and UWB positioning data, the accuracy of the recognition result is improved, and in a scenario with multiple persons, the following target can be effectively recognized, thereby realizing accurate following of the robot. Description of the Drawings

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required to be used in the embodiments of the present application will be briefly introduced below. For those of ordinary skill in the art, other drawings can also be obtained according to these drawings without creative efforts.

[0047] Figure 1 is a schematic flowchart of a method for robot following provided by an embodiment of the present application;

[0048] Figure 2 is a schematic flowchart of a similarity calculation method provided by an embodiment of the present application;

[0049] Figure 3 It is a schematic flowchart of a method for determining point position information provided by an embodiment of the present application;

[0050] Figure 4 It is an exemplary schematic diagram of a robot following method provided by an embodiment of the present application;

[0051] Figure 5 It is an exemplary schematic diagram of another robot following method provided by an embodiment of the present application;

[0052] Figure 6 It is a schematic structural diagram of a device for robot following provided by an embodiment of the present application;

[0053] Figure 7 It is a schematic structural diagram of an electronic device provided by another embodiment of the present application. Detailed implementation manners

[0054] The features and exemplary embodiments of various aspects of the present application will be described in detail below. To make the objectives, technical solutions, and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than limiting the present application. For those skilled in the art, the present application can be implemented without some of these specific details. The following description of the embodiments is only intended to provide a better understanding of the present application by showing examples of the present application.

[0055] It should be noted that, in this document, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover a non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising..." does not exclude the existence of additional identical elements in the process, method, article or device comprising the element.

[0056] To solve the problems existing in the prior art, embodiments of the present application provide a method, device, equipment, storage medium, and program product for robot following. First, based on Figure 1 A method for robot following provided by an embodiment of the present application will be introduced. As Figure 1 shown, this method is applied to an intelligent robot, and the method includes:

[0057] S101. Obtain the image to be recognized, the tracking linked list corresponding to the target to be followed, and the ultra-wideband (UWB) positioning data of the target to be followed.

[0058] Among them, the tracking linked list includes the predicted appearance features and predicted pose information of the target to be followed.

[0059] A UWB base station, a red, green, blue (RGB) camera, and a depth sensor are deployed on the intelligent robot. A UWB tag is placed on the target to be followed.

[0060] Specifically, the intelligent robot can collect RGB images within a 120° horizontal field of view in front of the intelligent robot through the RGB camera deployed in the front, and obtain the distance information and deflection angle information between the intelligent robot and the target to be followed, that is, the above-mentioned UWB positioning data, through the dual-antenna UWB base station and the UWB tag. The lidar deployed in the front of the intelligent robot can obtain 3D point clouds with a 360° view angle.

[0061] The tracking linked list includes the corresponding relationship between the appearance features and pose information of the target to be followed in each frame of the image. It should be noted that for each person in the image to be recognized, a corresponding tracking linked list can be established, and the tracking linked list stores the corresponding relationship between the appearance features and pose information of each person in each frame of the image.

[0062] Among them, for the tracking linked list corresponding to the target to be followed, the neural network model can predict the predicted appearance features and predicted pose information of the target to be followed in the next frame of the image based on the appearance features and pose information in the tracking linked list.

[0063] S102. For the image content of each person in the image to be recognized, obtain the appearance features and pose information of the person in the image content.

[0064] Among them, the detection boxes corresponding to each person in the image to be recognized are identified through the target detection model. Then, for the image content corresponding to each detection box, the appearance features and pose information of the person are obtained. Specifically, the pose information includes the position information and posture information of the person.

[0065] S103. Calculate the fusion evaluation results of the target to be followed and each person in the image content according to the appearance features and pose information of the person in the image content, the predicted appearance features, predicted pose information of the preset target to be followed, and the UWB positioning data.

[0066] Among them, the fusion evaluation results are used to measure the probability that the person in the image to be recognized is the target to be followed.

[0067] S104. When the maximum fusion evaluation result is greater than the preset threshold value, use the person corresponding to the maximum fusion evaluation result as the following target.

[0068] When the fusion evaluation results corresponding to all persons are less than or equal to the preset threshold value, it is determined that the tracking fails, and the point position information of the following target is obtained again, and the following target is located according to the obtained point position information.

[0069] S105. Follow the following target according to the positioning information of the following target.

[0070] Among them, the intelligent robot can collect RGB images through an RGB camera, collect three-dimensional point cloud information through a lidar, and collect the distance information and deflection angle information between the following target and the intelligent robot through a UWB base station and a UWB tag. Furthermore, through the calibration information of the lidar and the camera, the alignment of the point cloud and the RGB image is realized, and then through the pose transformation matrix, the positioning information of the following target is calculated.

[0071] By adopting the above method, the robot obtains the tracking linked list corresponding to the following target of the image to be recognized and the ultra-wideband UWB positioning data of the following target. Among them, the tracking linked list includes the predicted appearance features and predicted pose information of the following target, and obtains the appearance features and pose information corresponding to each person in the image to be recognized. Based on the predicted appearance features and predicted pose information of the following target, the fusion evaluation results of the predicted appearance features and predicted pose information and the appearance features and pose information of the persons in the image to be recognized can be calculated, so as to determine the following target in the image to be recognized. In this way, based on the appearance features and pose information, the following target in the image to be recognized can be judged from multiple dimensions, the accuracy of the recognition result is improved, and in a scene with multiple persons, the following target can be effectively recognized, so as to realize the accurate following of the robot.

[0072] In some embodiments of the present application, after obtaining the image to be recognized, a detection model is used to recognize the person images included in the image to be recognized and the pose information corresponding to each person, and then a re-identification model is used to obtain the appearance features of the person. Specifically, the above-mentioned obtaining the appearance features and pose information of the person in the image content can be realized as: inputting the image content into the re-identification model, and using the re-identification model to extract features from the image content to obtain appearance features. Input the image content into the detection model to obtain pose information.

[0073] It should be noted that for the following target, the above re-identification model can also be used to predict the appearance features of the following target in the next frame of the image to be recognized, and the detection model can also be used to predict the pose information of the following target in the next frame of the image to be recognized. That is, before obtaining the tracking linked list corresponding to the following target, the intelligent robot obtains a target image containing the following target; inputs the target image into the re-identification model to obtain the predicted appearance features; inputs the target image into the detection model to obtain the predicted pose information; constructs a tracking linked list containing the corresponding relationship between the predicted appearance features and the predicted pose information.

[0074] Among them, for the following target, multiple images containing only the following target are obtained, and these multiple images belong to a video stream collected by an RGB camera. In this way, the multiple images are sequentially input into the re-identification model according to the time stamp order, and the re-identification model can extract the appearance features of the following target in each image. The multiple images are sequentially input into the detection model according to the time order, and the detection model can extract the pose information of the following target in each image. Furthermore, the intelligent robot can determine the pose information corresponding to each appearance feature according to the time stamp, construct the corresponding relationship between the appearance features and the pose information, and obtain the tracking table corresponding to the following target.

[0075] In addition, the re-identification model can predict the appearance features of the following target in the next frame of the image based on the appearance features in the tracking table. The detection model includes a target detection network and a tracking network. The tracking network in the detection model can predict the pose information of the following target in the next frame of the image based on the pose information in the tracking table.

[0076] It should be noted that for each person, the intelligent robot can use the re-identification model and the detection model to construct a tracking table corresponding to each person.

[0077] In this way, the intelligent robot can use the collected target image containing the following target to extract the appearance features and pose information of the following target, and then predict the predicted appearance features and predicted pose information based on the extracted appearance features and pose information. In addition, the appearance features of each person in the image to be recognized can be extracted using the re-identification model, and the pose information of each person can be extracted using the detection model. In this way, the following target in the image to be recognized can be accurately determined through the fusion evaluation results corresponding to each person in the image to be recognized.

[0078] In the actual implementation process, for the above target detection network, the target detection network can be the yolo v8 target detection network. Specifically, based on the standard datasets VOC and COCO datasets, a training dataset containing person label data is constructed. The training data is training images containing only person images. The training images are input into the target detection network, and the detection boxes of each person contained in the training images output by the target detection network are obtained.

[0079] Furthermore, an enhanced training dataset is obtained, and the target detection network is trained using the enhanced dataset. Specifically, training images with occlusion, training images without occlusion, and training images with field of view exceeding are collected, and the above training images are input into the target detection network for fine-tuning training of the target detection network.

[0080] For the above re-identification model, the re-identification model can be obtained by training the CLIPReid network. Specifically, based on the standard dataset MSMT17 dataset, a training dataset is constructed. The training dataset includes the training image content of specific target persons. The training image content is input into the CLIPReid network, and the CLIPReid network infers the appearance features of the target person.

[0081] For the above tracking network, the tracking network can be trained based on the multi-object tracking algorithm of bortSort. After the intelligent robot inputs the image to be recognized into the detection model, the detection model obtains the detection box information of each person in the image to be recognized, and then crops the image content corresponding to each person according to the standard box. The image content of each person is pushed into the tracking network, and the tracking network infers the pose information of the person.

[0082] In some embodiments of the present application, as Figure 2 shown, for S103 above, according to the appearance features and pose information of the persons in the image content, as well as the predicted appearance features, predicted pose information, and UWB positioning data of the preset following target, calculate the fusion evaluation result of the following target and each person in the image content, which can be specifically implemented as:

[0083] S1031. Calculate the similarity evaluation result between the predicted appearance feature and the appearance feature of each person.

[0084] For the appearance feature of each person, calculate the cosine similarity between the predicted appearance feature and the appearance feature of each person respectively to obtain the similarity evaluation result.

[0085] In an example, Dsim(n)=cos_sim(Ftarget,Fdet(n)), n = 1, 2,..., N, where N represents the number of persons in the image to be recognized.

[0086] Among them, Dsim(n) represents the similarity evaluation result between the predicted appearance feature and the appearance feature of the nth person, Ftarget represents the predicted appearance feature, and Fdet(n) represents the appearance feature of the nth person.

[0087] S1032. Determine the predicted detection box position information of the following target according to the predicted pose information, and determine the detection box position information of each person according to the pose information of each person.

[0088] Among them, the predicted detection box position information includes the position of the detection box of the following target, including the positions of the four vertices of the detection box. The detection box position information of each person includes the positions of the four vertices of the detection box of each person.

[0089] S1033. Based on the predicted detection box position information and the detection box position information of each person, calculate the intersection over union evaluation result of the predicted detection box of the following target and the detection box of each person.

[0090] Among them, the intelligent robot determines the area of each person's detection box according to the positions of the four vertices of each detection box, calculates the overlapping area between the detection box of the following target and the detection box of each person, and the ratio to the area of the detection box of the following target, to obtain the intersection over union evaluation result corresponding to each person.

[0091] S1034. Determine the predicted center point position corresponding to the following target based on the UWB positioning data, and determine the center point position corresponding to each person based on the detection box position information of each person.

[0092] Among them, by performing coordinate transformation on the UWB positioning data, the predicted center point position of the following target in the image is determined.

[0093] S1035. Calculate the position deviation evaluation result based on the predicted center point position and the center point position of each person.

[0094] Among them, the intelligent robot obtains the position deviation evaluation result corresponding to each person by calculating the Euclidean distance between the predicted center point position and the center point position of each person in the image to be recognized.

[0095] S1036. Perform weighted summation on the similarity evaluation result, the intersection over union evaluation result, and the position deviation evaluation result to obtain the fusion evaluation result.

[0096] Among them, the similarity evaluation result, the intersection over union evaluation result, and the position deviation evaluation result respectively correspond to a preset weight, and the preset weight is set according to experience.

[0097] Specifically, the fusion evaluation result can be calculated according to the following formula:

[0098] Dtotal(n)=αDsim(n)+βDiou(n)+γDuwb(n);

[0099] Among them, Dtotal(n) represents the fusion evaluation result corresponding to the nth person, Dsim(n) represents the similarity evaluation result corresponding to the nth person, Diou(n) represents the intersection over union evaluation result corresponding to the nth person, and Duwb(n) represents the position deviation evaluation result corresponding to the nth person. α, β, and γ represent preset weights. In the embodiments of the present application, α is set to 0.8, and both β and γ are 0.5.

[0100] After obtaining the fusion evaluation result through the above calculation, the detection box of the following target determined by the intelligent robot is: max_id = Argmax(Dtotal). Where max_id refers to the detection box index corresponding to the maximum value of the fusion evaluation result.

[0101] bbox target =

[0102] Among them, bbox target represents the position of the detection box, bbox target = [x1, y1, x2, y2], where x1 and y1 represent the position of the upper left vertex of the detection box, and x2 and y2 represent the position of the lower right vertex of the detection box, with the unit being pixels. represents a preset threshold value.

[0103] By using the method provided in the embodiments of the present application, the intelligent robot respectively judges the fusion evaluation results of the following target and each person in the image to be recognized for appearance features, position information, and center point position, and comprehensively calculates the fusion evaluation results from multiple dimensions, improving the accuracy of the calculation results.

[0104] Based on the fusion evaluation results obtained for each person through the above calculation, the person corresponding to the following target in the image to be recognized can be determined according to the fusion evaluation results. Furthermore, the distance information of the following target is obtained through the UWB base station and the lidar carried by the intelligent robot, and the distance information is further processed to calculate the positioning information of the following target. Specifically, as Figure 3 shown, before S105 above, following the following target according to the positioning information of the following target, the method further includes:

[0105] S301. Obtain the point cloud distance information of the point cloud corresponding to the following target.

[0106] Among them, the intelligent robot obtains the point cloud falling within the detection box according to the detection box of the following target, and obtains the point cloud distance information of the above point cloud.

[0107] S302. Obtain the target point cloud information corresponding to the target point cloud according to the point cloud distance information and the preset distance ratio.

[0108] Among them, the intelligent robot determines the distance of each point cloud according to the point cloud distance information, then sorts the point clouds according to the distance, and obtains the target point cloud from the sorted point clouds according to a preset distance ratio.

[0109] S303. Calculate the average distance of the target point cloud based on the target point cloud information to obtain the detection distance of the following target.

[0110] To avoid the influence of noise, the point clouds within the detection frame can be removed according to a certain ratio. Specifically, according to the external parameters of the lidar and the camera and the projection matrix of the camera, the point clouds are projected into the camera coordinate system, and the point clouds falling within the detection frame are selected according to the detection frame. Each point cloud is represented as a three-dimensional vector (x, y, z), where x and y represent the horizontal and vertical coordinates in the camera coordinate system respectively, and z represents the depth value corresponding to the point cloud, and the depth value is the distance of the point cloud. The point clouds are sorted in ascending order according to the depth value, and 5% of the smallest point clouds are removed from the sorted point clouds. Then, 10% of the smallest point clouds are selected from the removed point clouds according to the depth value, and then the average distance of the selected point clouds is calculated to obtain the detection target.

[0111] S304. Perform Kalman filtering on the UWB positioning data and the detection distance to obtain the positioning information.

[0112] Specifically, the positioning information is calculated according to the following formula:

[0113] (xm, ym, zm) = KF((xu, yu, zu), (xd, yd, zd));

[0114] Among them, (xm, ym, zm) represents the positioning information, KF represents Kalman filtering, (xu, yu, zu) represents the UWB positioning data, and (xd, yd, zd) represents the detection distance.

[0115] By using the method provided in the embodiment of the present application, after obtaining the point cloud distance information and the measurement distance of the following target, in order to avoid the influence of noise, the point clouds can be screened according to a preset distance ratio to obtain the target point cloud information corresponding to the target point cloud. After calculating the detection distance of the target point cloud based on the target point cloud information, performing Kalman filtering on the detection distance and the UWB positioning data can achieve high-precision positioning of the following target.

[0116] It should be noted that in the case where the intelligent robot fails to follow the following target, the intelligent robot can play a preset voice to prompt that the following target positioning fails. After playing the preset voice, it starts to prepare for sound pickup. After receiving the audio data containing the preset wake-up word information, it calculates the position information of the following target according to the audio data; and then follows the following target according to the position information.

[0117] Specifically, the robot calculates the position of the following target based on the phase deviation of the sound source, thereby controlling the movement or turning of the robot to achieve re-following.

[0118] In addition, if the robot does not receive a sound source response, it remains stationary and continuously issues a tracking anomaly warning.

[0119] In this way, in the scenario where the intelligent robot loses tracking, the method of voice interaction is used to timely remind that the target following fails, and based on the re-tracking of sound source localization, following can be established according to the position information of the sound source.

[0120] The following combines Figure 4 to introduce the complete process of the above robot following, as Figure 4 shown, this method includes:

[0121] S401. Collect video through an RGB camera.

[0122] Specifically, an RGB camera is deployed in the front of the intelligent robot, and through this camera, video images within a 120° viewing angle in the front can be collected.

[0123] S402. Decode the video.

[0124] S403. Use a target detection network to detect pedestrians in the video images.

[0125] Among them, the video images are the images to be recognized in the above embodiments.

[0126] S404. Perform pedestrian re-identification on the person.

[0127] Specifically, the image content corresponding to each person in the image to be recognized obtained above is input into the re-identification model to obtain the appearance features corresponding to each person.

[0128] S405. Obtain UWB information.

[0129] Among them, a UWB base station is deployed in the front of the intelligent robot, and a UWB base station is deployed on the following target. Therefore, the intelligent robot can collect the distance information and deviation angle information of the following target, that is, the positioning data of the following target. The above UWB information is the positioning data of the following target.

[0130] S406. Determine the following target.

[0131] Regarding the above S406. Determine the following target, as Figure 5 shown, it can be specifically implemented as:

[0132] S4061. Analyze the tracking chain.

[0133] Among them, the tracking chain is the tracking chain corresponding to the target to be followed. The tracking chain includes the appearance features and pose information corresponding to the target to be followed in each frame of video image. And based on the historically stored appearance features and pose information in the tracking chain, the predicted appearance features and predicted pose information of the target to be followed in the image to be recognized can be predicted.

[0134] S4062. Analyze the UWB data.

[0135] Among them, the UWB data is the above-mentioned UWB positioning data. By analyzing the UWB data, the intelligent robot can calculate the predicted center point position of the target to be followed, and then calculate the position deviation evaluation result between the predicted center point position and the center point position of each person. The specific calculation method refers to the relevant description in the above-mentioned embodiment and will not be elaborated here.

[0136] S4063. Calculate the IOU based on the detection box.

[0137] Among them, the detection box is the detection box in the above-mentioned embodiment. The specific method for calculating the IOU refers to the relevant description in the above-mentioned embodiment and will not be elaborated here.

[0138] S4064. Calculate the similarity based on the ReID features.

[0139] Among them, after inputting the image content corresponding to each person in the image to be recognized into the re-identification model, the re-identification model can output the matching degree between this person and each person in the person database. The person database pre-stores the ReID and appearance features of each person. According to the matching degree output by the re-identification model, the ReID of the person corresponding to the highest matching degree can be determined, and then by querying the person database, the appearance features corresponding to this ReID can be obtained.

[0140] In addition, the re-identification model can also directly output the appearance features corresponding to each person.

[0141] After obtaining the appearance features corresponding to each person as described above, the intelligent robot calculates the similarity evaluation result between the predicted appearance features of the target to be followed and the appearance features corresponding to each person in the image to be recognized. The specific method for calculating the similarity evaluation result refers to the relevant description in the above-mentioned embodiment and will not be elaborated here.

[0142] S4065. Decide on the target to be followed.

[0143] Among them, the method for the intelligent robot to decide on the target to be followed is the method for determining the target to be followed according to the fusion evaluation result in the above-mentioned embodiment. The specific content refers to the relevant description in the above-mentioned embodiment and will not be elaborated here.

[0144] S4066. Maintain the tracking chain.

[0145] After determining the following target, the intelligent robot directly obtains the appearance features and pose information of the following target in the image to be recognized, and then updates the tracking chain of the following target by using the actually collected appearance features and pose information.

[0146] S407. Obtain point cloud information through lidar.

[0147] S408. Estimate the pose of the following target.

[0148] Specifically, the method for determining the positioning information of the following target according to the point cloud refers to the relevant description in the above embodiment, which will not be elaborated here.

[0149] After maintaining the tracking chain using the decision result above, S408 above, estimate the pose of the following target, as Figure 5 shown, can be specifically implemented as:

[0150] S4081. Determine whether the following is lost.

[0151] If so, execute S4087; if not, execute S4082.

[0152] S4082. Estimate the direction.

[0153] S4083. Estimate the distance.

[0154] Specifically, the intelligent robot can determine the corresponding method and distance of the following target based on the collected image and UWB positioning data.

[0155] S4084. Perform coordinate conversion.

[0156] S4085. Obtain distance information based on radar data.

[0157] S4086. Estimate the pose.

[0158] Specifically, the method for estimating the pose using radar data refers to the method for determining the position of the following target through point cloud above.

[0159] S4087. Analyze the UWB data.

[0160] S4088. Estimate the pose based on the UWB analysis data.

[0161] S409. Determine whether the pose is abnormal.

[0162] If so, execute S412; if not, execute S410.

[0163] S410. Perform motion control based on the pose information.

[0164] After the robot obtains the positioning information of the following target, it converts the relative position relationship from the camera coordinate system to the robot center coordinate system, and then uses the PID control algorithm to control the angular velocity and linear velocity of the robot, and sends the control strategy to the motion controller of the robot.

[0165] S411. The robot follows and executes.

[0166] S412. Perform voice alarm.

[0167] S413. Locate through the sound source.

[0168] By using the method provided in the embodiment of the present application, the UWB positioning data of the following target is obtained through the UWB sensor, and the image information and distance information are obtained through the camera and the depth sensor; then the detection model is used to detect each person in the image to be recognized, and the detection model and the re-identification model are used, and the UWB positioning data is fused to realize the recognition and tracking of the following target; then, based on the detection frame of the following target, the point cloud distance information collected by the lidar and the UWB positioning data are used to realize the positioning of the target, and according to the positioning information of the following target, the robot is controlled to perform motion following. In this way, accurate recognition and tracking of the following target can be realized in a multi-person scenario. In addition, when an abnormality occurs during tracking, re-tracking of the following target is realized through voice interaction and sound source positioning, improving the following robustness.

[0169] Based on the same concept, the embodiment of the present application further provides a device for robot following, which is applied to an intelligent robot, such as Figure 6 shown, the device includes:

[0170] An acquisition module 601, configured to acquire an image to be recognized, a tracking linked list corresponding to a following target, and the ultra-wideband UWB positioning data of the following target, where the tracking linked list includes the predicted appearance features and predicted pose information of the following target;

[0171] The acquisition module 601 is further configured to acquire the appearance features and pose information of the person in the image content for each person in the image to be recognized;

[0172] A calculation module 602, configured to calculate a fusion evaluation result of the following target and each person in the image content according to the appearance features and pose information of the person in the image content, the predicted appearance features, predicted pose information of a preset following target, and the UWB positioning data, where the fusion evaluation result is used to measure the probability that the person in the image to be recognized is the following target;

[0173] A determination module 603, configured to, when the maximum fusion evaluation result is greater than a preset threshold, use the person corresponding to the maximum fusion evaluation result as the following target;

[0174] A following module 604, configured to follow the following target according to the positioning information of the following target.

[0175] In a possible implementation, the acquisition module 601 is specifically configured to:

[0176] Input the image content into a re-identification model, and use the re-identification model to extract features from the image content to obtain the appearance features;

[0177] Input the image content into a detection model to obtain the pose information.

[0178] In a possible implementation, the device further includes:

[0179] The acquisition module 601 is further configured to acquire a target image including the following target;

[0180] A prediction module, configured to input the target image into a re-identification model to obtain the predicted appearance features;

[0181] The prediction module is further configured to input the target image into a detection model to obtain the predicted pose information;

[0182] A construction module, configured to construct a tracking linked list including the correspondence between the predicted appearance features and the predicted pose information.

[0183] In a possible implementation, the calculation module 602 is specifically configured to:

[0184] Calculate the similarity evaluation result between the predicted appearance features and the appearance features of each person;

[0185] Determine the predicted detection box position information of the following target according to the predicted pose information, and determine the detection box position information of each person according to the pose information of each person;

[0186] Based on the predicted detection box position information and the detection box position information of each person, calculate the intersection-over-union evaluation result between the predicted detection box of the following target and the detection box of each person;

[0187] Determine the predicted center point position corresponding to the following target based on the UWB positioning data, and determine the center point position corresponding to each person based on the detection box position information of each person;

[0188] Calculate the position deviation evaluation result based on the predicted center point position and the center point position of each person;

[0189] Perform a weighted sum of the similarity evaluation result, the intersection over union evaluation result, and the position deviation evaluation result to obtain the fusion evaluation result.

[0190] In a possible implementation, the device further includes:

[0191] An acquisition module 601, further configured to acquire the point cloud distance information of the point cloud corresponding to the following target;

[0192] The acquisition module 601 is further configured to acquire the target point cloud information corresponding to the target point cloud according to the point cloud distance information and a preset distance ratio;

[0193] A calculation module 602, further configured to calculate the distance mean value of the target point cloud based on the target point cloud information to obtain the detection distance of the following target;

[0194] A filtering module, configured to perform Kalman filtering on the UWB positioning data and the detection distance to obtain the positioning information.

[0195] In a possible implementation, the device further includes:

[0196] A receiving module, configured to receive audio data including preset wake-up word information when it is determined that the following of the following target fails;

[0197] The calculation module 602 is further configured to calculate the position information of the following target according to the audio data;

[0198] A following module 604, configured to follow the following target according to the position information.

[0199] It should be noted that the device for robot following corresponds to the method for robot following applied to an intelligent robot. All implementation manners in the above method embodiments are applicable to the embodiments of this device and can achieve the same technical effects.

[0200] Figure 7 FIG. shows a schematic hardware structure diagram of an electronic device provided in an embodiment of the present application.

[0201] The electronic device may include a processor 701 and a memory 702 storing computer program instructions.

[0202] Specifically, the above-mentioned processor 701 may include a Central Processing Unit (CPU), or an Application Specific Integrated Circuit (ASIC), or may be configured as one or more integrated circuits for implementing the embodiments of the present application.

[0203] The memory 702 may include a mass storage for data or instructions. By way of example and not limitation, the memory 702 may include a Hard Disk Drive (HDD), a floppy disk drive, a flash memory, an optical disc, a magneto-optical disc, a magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. In a suitable case, the memory 702 may include removable or non-removable (or fixed) media. In a suitable case, the memory 702 may be internal or external to the integrated gateway disaster recovery device. In a specific embodiment, the memory 702 is a non-volatile solid-state memory.

[0204] The memory may include a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk storage media device, an optical storage media device, a flash memory device, an electrical, optical, or other physical / tangible memory storage device. Thus, generally, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to one aspect of the present disclosure.

[0205] The processor 701 reads and executes the computer program instructions stored in the memory 702 to implement any one of the privacy set intersection methods in the above embodiments.

[0206] In one example, the electronic device may further include a communication interface 703 and a bus 704. Among them, as Figure 7 shown, the processor 701, the memory 702, and the communication interface 703 are connected through the bus 704 to complete communication with each other.

[0207] The communication interface 703 is mainly used to implement communication between each module, device, unit, and / or device in the embodiments of the present application.

[0208] The bus 704 includes hardware, software, or both, and couples components of an electronic device to each other. By way of example and not limitation, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Super Transmission (HT) interconnect, an Industrial Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low PinCount (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses or a combination of two or more of these. Where appropriate, the bus 704 may include one or more buses. Although the embodiments of the present application describe and illustrate specific buses, the present application contemplates any suitable bus or interconnect.

[0209] In addition, in combination with the method for robot following in the above embodiments, an embodiment of the present application may provide a computer storage medium for implementation. Computer program instructions are stored on the computer storage medium; when the computer program instructions are executed by a processor, any one of the methods for robot following in the above embodiments is implemented.

[0210] In combination with the method for robot following in the above embodiments, an embodiment of the present application may provide a computer program product. When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device is caused to execute any one of the methods for robot following in the above embodiments.

[0211] It should be clear that the present application is not limited to the specific configurations and processes described above and illustrated in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and illustrated as examples. However, the method process of the present application is not limited to the specific steps described and illustrated, and those skilled in the art can make various changes, modifications, and additions, or change the order between steps after understanding the spirit of the present application.

[0212] The functional blocks shown in the above structural block diagrams can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application-specific integrated circuit (ASIC), appropriate firmware, a plug-in, a functional card, and so on. When implemented in software, the elements of the present application are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium or transmitted via a data signal carried in a carrier wave over a transmission medium or a communication link. A "machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable read-only memory (EROM), floppy disks, compact disc read-only memory (CD-ROM), optical discs, hard disks, fiber optic media, radio frequency (RF) links, and so on. The code segment can be downloaded via a computer network such as the Internet, an intranet, and so on.

[0213] It should also be noted that the exemplary embodiments mentioned in the present application describe some methods or systems based on a series of steps or devices. However, the present application is not limited to the order of the above steps, that is, the steps can be executed in the order mentioned in the embodiments, or different from the order in the embodiments, or several steps can be executed simultaneously.

[0214] Aspects of the present disclosure have been described above with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block in the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions executed by the processor of the computer or other programmable data processing apparatus enable the implementation of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It should also be understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by dedicated hardware that performs the specified functions or acts, or can be implemented by a combination of dedicated hardware and computer instructions.

[0215] As described above, the foregoing is only a specific implementation manner of the present application. Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, modules, and units described above can refer to the corresponding processes in the foregoing method embodiments and will not be described herein again. It should be understood that the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or substitutions within the technical scope disclosed in the present application, and these modifications or substitutions should all be covered by the protection scope of the present application.

Claims

1. A method for robot following, characterized in that, Applied to an intelligent robot, the method includes: Obtaining an image to be recognized, a tracking linked list corresponding to a following target, and ultra-wideband UWB positioning data of the following target, where the tracking linked list includes predicted appearance features and predicted pose information of the following target; For the image content of each person in the image to be recognized, obtaining the appearance features and pose information of the person in the image content; According to the appearance features and pose information of the person in the image content, as well as the predicted appearance features, predicted pose information of a preset following target, and the UWB positioning data, calculating a fusion evaluation result between the following target and each person in the image content, where the fusion evaluation result is used to measure the probability that the person in the image to be recognized is the following target; When the maximum fusion evaluation result is greater than a preset threshold, taking the person corresponding to the maximum fusion evaluation result as the following target; Following the following target according to the positioning information of the following target; The obtaining the appearance features and pose information of the person in the image content includes: Inputting the image content into a re-identification model, and using the re-identification model to extract features from the image content to obtain the appearance features, where the re-identification model is obtained by training a CLIPReid network; Inputting the image content into a detection model to obtain the pose information, where the detection model includes an object detection network and a tracking network, the object detection network is used to identify the person in the image content, and the tracking network is used to predict the pose information of the following target in the next frame of image based on the pose information in the tracking linked list; The calculating the fusion evaluation result between the following target and each person in the image content according to the appearance features and pose information of the person in the image content, as well as the predicted appearance features, predicted pose information of a preset following target, and the UWB positioning data includes: Calculating a similarity evaluation result between the predicted appearance features and the appearance features of each person; Determining the position information of the predicted detection box of the following target according to the predicted pose information, and determining the position information of the detection box of each person according to the pose information of each person; Based on the position information of the predicted detection box and the position information of the detection box of each person, calculating an intersection over union evaluation result between the predicted detection box of the following target and the detection box of each person; Based on the UWB positioning data, determining the predicted center point position corresponding to the following target, and based on the position information of the detection box of each person, determining the center point position corresponding to each person; Calculating a position deviation evaluation result based on the predicted center point position and the center point position of each person; Performing weighted summation on the similarity evaluation result, the intersection over union evaluation result, and the position deviation evaluation result to obtain the fusion evaluation result.

2. The method according to claim 1, characterized in that, Before the obtaining the image to be recognized, the tracking linked list corresponding to the following target, and the ultra-wideband UWB positioning data of the following target, the method further includes: Obtaining a target image including the following target; Inputting the target image into a re-identification model to obtain the predicted appearance features; Input the target image into the detection model to obtain the predicted pose information; Construct a tracking linked list that contains the correspondence between the predicted appearance features and the predicted pose information.

3. The method according to claim 1 or 2, characterized in that, Before following the following target according to the positioning information of the following target, the method further includes: Obtain the point cloud distance information of the point cloud corresponding to the following target; Obtain the target point cloud information corresponding to the target point cloud according to the point cloud distance information and a preset distance ratio; Calculate the distance mean of the target point cloud based on the target point cloud information to obtain the detection distance of the following target; Perform Kalman filtering on the UWB positioning data and the detection distance to obtain the positioning information.

4. The method according to claim 1, characterized in that, After following the following target according to the positioning information of the following target, the method further includes: In the case of determining that the following of the following target fails, receive audio data containing preset wake-up word information; Calculate the position information of the following target according to the audio data; Follow the following target according to the position information.

5. A device for robot following, characterized in that, Applied to an intelligent robot, the device includes: An acquisition module, configured to acquire an image to be recognized, a tracking linked list corresponding to a following target, and ultra-wideband UWB positioning data of the following target, where the tracking linked list includes predicted appearance features and predicted pose information of the following target; The acquisition module is further configured to acquire the appearance features and pose information of the person in the image content for the image content of each person in the image to be recognized; A calculation module, configured to calculate a fusion evaluation result of the following target and each person in the image content according to the appearance features and pose information of the person in the image content, the predicted appearance features, predicted pose information of a preset following target, and the UWB positioning data, where the fusion evaluation result is used to measure the probability that the person in the image to be recognized is the following target; A determination module, configured to use the person corresponding to the maximum fusion evaluation result as the following target when the maximum fusion evaluation result is greater than a preset threshold; A following module, configured to follow the following target according to the positioning information of the following target; The acquisition module is specifically configured to: Input the image content into a re-identification model, and use the re-identification model to extract features from the image content to obtain the appearance features, where the re-identification model is obtained by training a CLIPReid network; Input the image content into a detection model to obtain the pose information, where the detection model includes a target detection network and a tracking network, the target detection network is used to identify the person in the image content, and the tracking network is used to predict the pose information of the following target in the next frame of image based on the pose information in the tracking linked list; The calculation module is specifically configured to: Calculate a similarity evaluation result between the predicted appearance features and the appearance features of each person; Determine the predicted detection frame position information of the following target according to the predicted pose information, and determine the detection frame position information of each person according to the pose information of each person; Based on the predicted detection box position information and the detection box position information of each person, calculate the intersection over union (IoU) evaluation result between the predicted detection box of the following target and the detection box of each person; Based on the UWB positioning data, determine the predicted center point position corresponding to the following target, and based on the detection box position information of each person, determine the center point position corresponding to each person; Calculate the position deviation evaluation result based on the predicted center point position and the center point position of each person; Perform a weighted sum of the similarity evaluation result, the intersection over union evaluation result, and the position deviation evaluation result to obtain the fusion evaluation result.

6. An electronic device, characterized in that, The device includes: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, it implements the robot following method according to any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, Computer program instructions are stored on the computer-readable storage medium, and when the computer program instructions are executed by the processor, they implement the robot following method according to any one of claims 1-4.

8. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device is caused to execute the robot following method according to any one of claims 1-4.

Images