Target following method and apparatus, movable following device and storage medium

Abstract

A target following method and apparatus, a movable following device and a storage medium. The target following method is used for the movable following device, and comprises: performing target detection processing and feature matching processing on an obtained target image, so as to obtain a target detection box corresponding to a following target; according to the target detection box and first coordinates corresponding to the following device, obtaining second coordinates of the following target; and updating historical navigation data of the following device according to the second coordinates so as to obtain target navigation data, and controlling the following device to move on the basis of the target navigation data.

Description

Target following method, device, movable following device and storage medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the Chinese patent application filed with the China Patent Office on August 22, 2023, with application number 202311067878.2 and invention name “Target following method, device, removable following device and storage medium”, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present application relates to the field of intelligent robot technology, and in particular to a target following method, apparatus, movable following device and storage medium. Background Art

[0004] With the rapid development of artificial intelligence and robotics, robots are able to interact with humans and perform various tasks. However, in practical applications, the humanization and interactivity of robots still face some challenges, one of which is how to accurately follow the movement of humans. Robots can be regarded as mobile following devices.

[0005] In the related art, target following methods used in movable following devices generally require setting a special marker on the following target first, so that the following device can determine the position information of the following target based on the special marker, thereby achieving target following.

[0006] However, the above target following methods have poor flexibility.

[0007] Summary of the Invention

[0008] Based on this, it is necessary to provide a target following method, apparatus, movable following device and readable storage medium with good flexibility to address the above technical problems.

[0009] In a first aspect, the present application provides a target following method for a movable following device, the method comprising:

[0010] Perform target detection and feature matching on the acquired target image to obtain a target detection frame corresponding to the target being followed;

[0011] Obtaining a second coordinate of the target to be followed based on the target detection frame and the first coordinate corresponding to the following device;

[0012] The historical navigation data of the following device is updated according to the second coordinate to obtain target navigation data, and the movement of the following device is controlled based on the target navigation data.

[0013] In one embodiment, target detection processing and feature matching processing are performed on the acquired target image to obtain a target detection frame corresponding to the follow target, including:

[0014] Perform target detection on the target image to obtain candidate detection frames and candidate features corresponding to the candidate targets;

[0015] Based on a sample feature library including sample features and sample identifiers corresponding to the sample features, matching the candidate features to obtain candidate identifiers corresponding to the candidate features;

[0016] If the candidate identifier is the same as the target identifier of the target being followed, the candidate detection frame corresponding to the candidate target is used as the target detection frame.

[0017] In one embodiment, before performing target detection and feature matching on the acquired target image to obtain a target detection frame corresponding to the follow target, the method further includes:

[0018] Continuously perform feature matching processing on the environmental images acquired by the following device to obtain the identification to be started;

[0019] For the pending start flag, detecting whether the pending start flag is obtained continuously for a preset time period;

[0020] If the to-be-activated mark is obtained continuously for a preset time period, the face angle corresponding to the to-be-activated mark is obtained;

[0021] If the face angle falls within the preset angle range, the marker to be activated will be used as the target marker.

[0022] In one embodiment, before performing target detection and feature matching on the acquired target image to obtain a target detection frame corresponding to the follow target, the method further includes:

[0023] If voice data containing the target wake-up word is detected, the voice data is processed for sound source localization to determine the direction to be activated;

[0024] Perform speech recognition processing on the voice data to obtain the usage intention corresponding to the voice data;

[0025] Obtain an environment image corresponding to the direction to be started, and perform target detection processing on the environment image to obtain a detection result to be started;

[0026] If the to-be-activated detection result indicates that the environment image includes at least one to-be-activated target, and the usage intention is a preset follow intention, the target identifier is determined based on the to-be-activated identifier corresponding to the to-be-activated target.

[0027] In one embodiment, continuously performing feature matching processing on the environment image acquired by the following device to obtain the to-be-started identifier includes:

[0028] Continuously perform feature extraction and processing on the environmental images acquired by the following device to obtain the features to be started;

[0029] The sample identifier corresponding to the sample feature that successfully matches the feature to be started among the multiple sample features in the sample feature library is used as the identifier to be started;

[0030] If the feature to be activated fails to match the sample feature in the sample feature library, a to-be-activated identifier is generated for the feature to be activated, and the to-be-activated identifier is used as the sample identifier and the feature to be activated is stored in the sample feature library as the sample feature.

[0031] In one embodiment, obtaining a second coordinate of the follow target according to the target detection frame and the first coordinate corresponding to the follow device includes:

[0032] Depth estimation is performed based on the target detection frame and the preset calibration frame to obtain the target distance between the tracking target and the tracking device;

[0033] The deflection angle is calculated based on the target detection frame and the image center point of the target image to obtain the target angle between the tracking target and the following device;

[0034] The second coordinate is obtained by calculating according to the target distance, the target angle and the first coordinate.

[0035] In one embodiment, the historical navigation data includes historical navigation paths and historical destination coordinates, and the target navigation data includes target navigation paths and target destination coordinates; updating the historical navigation data of the following device according to the second coordinates to obtain the target navigation data includes:

[0036] The second coordinate is used as a new path point to update the historical navigation path to obtain the target navigation path;

[0037] Calculate the distance between the second coordinate and the historical target coordinate to obtain the distance difference;

[0038] If the distance difference is greater than the difference threshold, determining the target destination coordinates based on the second coordinates;

[0039] If the distance difference is less than or equal to the difference threshold, the historical destination coordinates are determined as the target destination coordinates.

[0040] In one embodiment, controlling the movement of the following device based on the target navigation data includes:

[0041] Generate movement instructions corresponding to the following device based on the target navigation data;

[0042] Perform local path planning based on the environmental point cloud data obtained by the following device, and obtain the corresponding avoidance instructions for the following device;

[0043] Control the movement of the follower device based on movement commands and avoidance commands.

[0044] In one embodiment, the method further comprises:

[0045] If the target cannot be detected for a preset period of time, a target loss command is generated.

[0046] In one embodiment, the method further comprises:

[0047] In response to the target lost instruction, determining a search center coordinate based on the second coordinate;

[0048] Determine an area of ​​a preset size as a first search area according to the search center coordinates;

[0049] Controlling the following device to traverse the first search area and continuously performing feature matching processing on the acquired search image until a first matching result indicates that a following target exists in the first search area, or the traversal ends;

[0050] If the first matching result indicates that there is a follow-up target in the first search area, an object detection frame corresponding to the follow-up target is obtained.

[0051] In one embodiment, the method further comprises:

[0052] During the execution of the target lost instruction, if voice data including the target wake-up word is detected, the voice data is subjected to sound source localization processing to determine a second search direction;

[0053] Controlling the following device to turn to a second search direction, and determining a second search area based on the second search direction and the first coordinates;

[0054] Controlling the following device to traverse the second search area, and continuously performing a feature matching process on the acquired search image until a second result indicates that a following target exists in the second search area, or the traversal ends;

[0055] If the second matching result indicates that there is a follow-up target in the second search area, an object detection frame corresponding to the follow-up target is obtained.

[0056] In a second aspect, the present application further provides a target following device for a movable following device, the device comprising:

[0057] The target detection module is used to perform target detection and feature matching on the acquired target image to obtain a target detection frame corresponding to the target being followed;

[0058] A coordinate determination module, configured to obtain a second coordinate of the follow target based on the target detection frame and the first coordinate corresponding to the follow device;

[0059] The navigation control module is used to update the historical navigation data of the following device according to the second coordinate, obtain target navigation data, and control the movement of the following device based on the target navigation data.

[0060] In a third aspect, the present application also provides a movable following device, characterized in that the device includes an image acquisition device, a voice acquisition device, a lidar sensor and a target following device; the target following device is used to implement the steps of the method described in the first aspect above.

[0061] In a fourth aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method described in the first aspect when executing the computer program.

[0062] In a fifth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0063] The above-mentioned target following method, device, movable following device and storage medium, the method is used for a movable following device, by performing target detection processing and feature matching processing on the acquired target image, obtaining a target detection frame corresponding to the following target; according to the target detection frame and the first coordinate corresponding to the following device, obtaining the second coordinate of the following target; according to the second coordinate, the historical navigation data of the following device is updated to obtain target navigation data, and the movement of the following device is controlled based on the target navigation data; in this way, by performing target detection processing and feature matching processing on the acquired target image, a target detection frame of the following target is obtained, and then the position information of the following target is determined based on the target detection frame and the current position information of the following device, the following device can follow any following target, avoiding the problem of poor flexibility caused by the need to set special markers on the following target side in traditional technology to obtain the position information of the following target through special markers. The target following method provided in this application has good flexibility. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] FIG1 is a schematic flow chart of a target following method according to an embodiment;

[0065] FIG2 is a schematic diagram of a process for obtaining a target detection frame in one embodiment;

[0066] FIG3 is a schematic diagram of a process for obtaining a target identification step in one embodiment;

[0067] FIG4 is a schematic diagram of a flow chart of the step of obtaining a to-be-started identifier in one embodiment;

[0068] FIG5 is a schematic flow chart of the step of obtaining a target identification in another embodiment;

[0069] FIG6 is a schematic flow chart of the step of obtaining the second coordinates in one embodiment;

[0070] FIG7 is a schematic flow chart of steps for obtaining target navigation data according to an embodiment;

[0071] FIG8 is a schematic diagram of a flow chart of steps for controlling movement of a follower device based on navigation data in one embodiment;

[0072] FIG9 is a schematic diagram of a flow chart of target search in response to a target lost instruction in one embodiment;

[0073] FIG10 is a flow chart of a target search process in response to a target loss instruction in another embodiment;

[0074] FIG11 is a schematic diagram of the mechanism of a target following device in one embodiment;

[0075] FIG12 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0076] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0077] In one embodiment, as shown in FIG1 , a target following method is provided. The method is described by taking the application of the method in a movable following device as an example, and includes the following steps:

[0078] Step 102: Perform target detection and feature matching on the acquired target image to obtain a target detection frame corresponding to the follow target.

[0079] The target image refers to the image captured by the following device during the process of starting the following.

[0080] Exemplarily, the following device is a quadruped robot, such as a robot dog; an image acquisition device, such as a monocular camera, is mounted on the front of the robot dog's head. The monocular camera is angled slightly upward to ensure that it can capture a relatively complete image of the entire body of a human of average height (exemplarily, 1.7 meters) at an appropriate distance (exemplarily, 0.5 meters to 1 meter). It should be noted that the image acquisition device is not limited to a monocular camera; any camera capable of acquiring RGB images is acceptable. This application does not specifically limit the device for acquiring the target image.

[0081] Among them, target detection processing on the target image refers to detecting the human body in the target image. Multiple candidate human targets may be detected, and features of each candidate target need to be extracted to determine the target detection frame corresponding to the follow target.

[0082] In some other embodiments, the object detection box may also be referred to as a bounding box or a bounding box.

[0083] Exemplarily, the target detection process may be performed using a FastestDet (Fastest Object Detection) model or a YOLO (You Only Look Once) series model.

[0084] Step 104 : Obtain the second coordinates of the target to be followed according to the target detection frame and the first coordinates corresponding to the following device.

[0085] The first coordinate refers to the world coordinate corresponding to the current position of the following device. For example, the target detection frame can be used to estimate the distance and deflection angle between the following target and the following device. Therefore, based on the target detection frame and the first coordinate, the world coordinate corresponding to the following target, i.e., the second coordinate, can be estimated. The second coordinate is the position of the following target when the target image is acquired.

[0086] Step 106 : updating the historical navigation data of the following device according to the second coordinate to obtain target navigation data, and controlling the movement of the following device based on the target navigation data.

[0087] The historical navigation data refers to the navigation data used by the following device to track the target before the target image is acquired. The historical navigation data is updated according to the second coordinates, and the obtained target navigation data can determine the latest position of the target.

[0088] The following device is controlled to move according to the target navigation data, that is, the following device is controlled to move toward the position corresponding to the second coordinate.

[0089] The target following method provided in the above embodiment is used for a movable following device. By performing target detection processing and feature matching processing on the acquired target image, a target detection frame corresponding to the following target is obtained; based on the target detection frame and the first coordinate corresponding to the following device, a second coordinate of the following target is obtained; based on the second coordinate, the historical navigation data of the following device is updated to obtain target navigation data, and the movement of the following device is controlled based on the target navigation data; in this way, by performing target detection processing and feature matching processing on the acquired target image, a target detection frame of the following target is obtained, and then the position information of the following target is determined based on the target detection frame and the current position information of the following device, the following device can follow any following target, avoiding the problem of poor flexibility caused by the need to set a special marker on the side of the following target in traditional technology to obtain the position information of the following target through the special marker. The target following method provided in this application has better flexibility. Optionally, by performing target detection processing and feature matching processing on the acquired target image, the target human body can be detected more accurately, and the human body features can be recorded for matching to ensure that the wrong person is not followed.

[0090] In one embodiment, referring to FIG2 , the process involves performing target detection and feature matching on the acquired target image to obtain a target detection frame corresponding to the target. As shown in FIG2 , the process includes:

[0091] Step 202: Perform target detection processing on the target image to obtain candidate detection boxes and candidate features corresponding to the candidate targets.

[0092] There may be one, multiple, or zero candidate targets. For example, if there are zero candidate targets, it means that the target image acquired by the following device during the following process does not contain any human targets, and the target may be lost.

[0093] Exemplarily, the candidate detection frame is based on the quadrilateral marked by the candidate target, that is, the position of the candidate target in the target image. The specific form can be the coordinate data of the four vertices of the quadrilateral relative to the target image, or the coordinate data and length and width data of the center point of the quadrilateral relative to the target image. The candidate feature is the feature data corresponding to the candidate target.

[0094] Step 204 : Based on the sample feature library including the sample features and the sample identifiers corresponding to the sample features, the candidate features are matched to obtain the candidate identifiers corresponding to the candidate features.

[0095] The candidate feature is matched with each sample feature in the sample feature library. If the match is successful, the sample identifier of the successfully matched sample data is used as the candidate identifier corresponding to the candidate feature.

[0096] For example, feature matching can be performed using a classification network; for another example, feature matching can be performed using the Euclidean distance or cosine similarity between the candidate feature and each sample feature. The present embodiment does not limit the specific method of feature matching.

[0097] Step 206: If the candidate identifier is the same as the target identifier of the target to be followed, the candidate detection frame corresponding to the candidate target is used as the target detection frame.

[0098] Among them, if the number of candidate targets is one, it is determined whether the candidate identifier of the candidate target is the same as the target identifier. If they are the same, the candidate target is considered to be the follow target, and the corresponding candidate detection frame is used as the target detection frame. If the number of candidate targets is multiple, the candidate detection frame corresponding to the candidate target whose candidate identifier is the same as the target identifier is determined as the target detection frame.

[0099] In a possible implementation, if all candidate identifiers are different from the target identifier of the follow target, it can be considered that the target image does not include the follow target, and no target detection frame will be output.

[0100] The above embodiment performs target detection processing on the target image to obtain candidate detection frames and candidate features corresponding to the candidate targets; based on a sample feature library including sample features and sample identifiers corresponding to the sample features, the candidate features are matched to obtain candidate identifiers corresponding to the candidate features; if the candidate identifier is the same as the target identifier of the followed target, the candidate detection frame corresponding to the candidate target is used as the target detection frame, which can improve the accuracy of the target detection frame and ensure that the following device will not follow the wrong target.

[0101] In one embodiment, referring to FIG3 , this embodiment relates to how a following device determines a target to be followed before performing target detection and feature matching on a captured target image to obtain a target detection frame corresponding to the target to be followed. As shown in FIG3 , this process includes:

[0102] Step 302: continuously perform feature matching processing on the environment image acquired by the following device to obtain a to-be-started identifier.

[0103] Before starting target following, the following device continuously acquires the environment image and performs feature matching processing on the environment image to determine whether a corresponding to-be-started marker exists in the environment image.

[0104] In one possible implementation, as shown in FIG4 , the process of continuously performing feature matching processing on the environment image acquired by the following device to obtain the to-be-started identifier includes:

[0105] Step 402 : continuously perform feature extraction processing on the environment image acquired by the following device to obtain features to be activated.

[0106] The feature to be started refers to feature data corresponding to the human target in the environment image acquired by the following device before starting the target following process.

[0107] Step 404: The sample identifier corresponding to the sample feature that successfully matches the feature to be started among the multiple sample features in the sample feature library is used as the identifier to be started.

[0108] The feature data to be started is matched with multiple sample features in the sample feature library to check whether the feature to be started is a human target recorded in the sample feature library. If so, the sample identifier corresponding to the successfully matched sample feature is used as the identifier to be started.

[0109] Step 406: If the feature to be activated fails to match the sample features in the sample feature library, a to-be-activated identifier is generated for the feature to be activated, and the to-be-activated identifier is used as the sample identifier and the feature to be activated is stored in the sample feature library as the sample feature.

[0110] If the feature to be activated is not a human target that has been recorded in the corresponding sample feature library, the feature to be activated is stored in the sample feature library, and a corresponding identifier to be activated is generated for the feature to be activated and stored in the sample feature library as a sample identifier.

[0111] In this way, the following device can initiate following processing for any human target, and is not limited to a specific human object.

[0112] In one possible implementation, the sample feature library categorizes sample features into temporary sample features and recorded sample features. Temporary sample features are obtained based on the method of step 406, while recorded sample features are features of the target person specifically recorded by the following device. It is understood that recorded sample features are sample features corresponding to a specific user of the following device, while temporary sample features are obtained based on environmental image detection and are not user-specific.

[0113] In this possible implementation, temporary sample features in the sample feature library are cleaned up regularly according to usage to prevent low matching efficiency during the target tracking process.

[0114] For example, if the temporary sample feature data has not been used within a preset timeout period, the data corresponding to the temporary sample feature will be deleted from the sample feature library. This can be determined by reading the call time corresponding to the temporary sample feature, or by other methods to determine whether the temporary sample feature has timed out and is not used. The preset timeout period can be one day or one hour, depending on the hardware configuration of the tracking device.

[0115] Step 304: for the to-be-started flag, detecting whether the to-be-started flag is obtained continuously for a preset time period.

[0116] Exemplarily, a general following device continuously acquires environmental images at a preset frequency, and detects whether the pending start indication is obtained in a consecutive preset number of environmental images in a manner that detects whether the pending start indication is obtained in a continuous preset number of images; another exemplary method is to record the appearance time of each pending start indication in the corresponding field when the pending start indication is obtained for the first time; if the appearance time does not meet the continuous rule, the field is cleared to zero; if the appearance time meets the continuous rule, the appearance time is recorded until the preset time length is met.

[0117] Step 306: If the to-be-activated marker is obtained continuously for a preset time period, the face angle corresponding to the to-be-activated marker is obtained.

[0118] The face angle refers to the angle of the target human body corresponding to the to-be-activated marker relative to the image acquisition device of the following device.

[0119] Exemplarily, when the to-be-started mark is obtained continuously for a preset period of time, the target human body corresponding to the start mark is obtained for face recognition, the facial feature data corresponding to the start mark is obtained and the facial angle is calculated to obtain the facial angle corresponding to the to-be-started mark.

[0120] For example, the face angle corresponding to the marker to be activated may be obtained by using solvePnP (solve Perspective-n-Point) pose estimation or 3DMM (3D Morphable Model) parameter face pose estimation.

[0121] Optionally, if the to-be-started mark cannot be obtained within a continuous preset time period, the operation of obtaining the face angle is not performed, and target detection and feature matching processing are continued on the environmental image.

[0122] Step 308: If the face angle belongs to the preset angle range, the marker to be started is used as the target marker.

[0123] The preset angle range is used to represent the angle range of a human face that can be considered to be facing the image acquisition device of the following device. If it is determined that the target human face corresponding to the to-be-activated identifier is facing the following device, the to-be-activated identifier is used as the target identifier, and the target tracking process is initiated.

[0124] Optionally, if the face angle does not fall within the preset angle range, the target following process is not started.

[0125] In the method provided in the above embodiment, when the following device determines the following target, the visual information of the following device is used to determine whether to start following. If the current human body appears in the field of view of the following device for a long enough time and the following target can detect that the face of the human body is facing the following device, the following device determines the target identifier based on the identifier corresponding to the human body and starts following the target identifier. In this way, the selection and switching of the following target can be easily performed during the human-computer interaction process, which is very flexible.

[0126] In one embodiment, referring to FIG5 , this embodiment relates to how a following device determines a target to be followed before performing target detection and feature matching on a captured target image to obtain a target detection frame corresponding to the target to be followed. As shown in FIG5 , this process includes:

[0127] Step 502: If voice data including the target wake-up word is detected, the voice data is processed for sound source localization to determine the direction to be activated.

[0128] The voice data can be acquired by a voice acquisition device mounted on the follower device. The voice acquisition device involved in this application can perform 360-degree sound source positioning, enhancement and separation processing. For example, the voice acquisition device can be a microphone array.

[0129] To determine whether the voice data includes the target wake-up word, automatic speech recognition (ASR) technology can be used.

[0130] Step 504: Perform speech recognition processing on the speech data to obtain the usage intention corresponding to the speech data.

[0131] Step 506 , obtaining an environment image corresponding to the direction to be started, and performing target detection processing on the environment image to obtain a detection result to be started.

[0132] Among them, obtaining the environmental image corresponding to the direction to be started includes controlling the following device to turn to the direction to be started according to the direction to be started, and then obtaining the environmental image in the current field of view.

[0133] Among them, target detection processing is performed on the environmental image to extract human targets included in the environmental image. At least one human target may be detected, that is, the to-be-started detection box corresponding to the at least one human target is output as the to-be-started detection result. It is also possible that no human target is detected, that is, the to-be-started detection result is empty data.

[0134] It should be noted that, in actual application scenarios, step 506 and step 504 are generally performed simultaneously.

[0135] Step 508: If the to-be-activated detection result indicates that the environment image includes at least one to-be-activated target, and the usage intention is a preset follow intention, the target identifier is determined based on the to-be-activated identifier corresponding to the to-be-activated target.

[0136] The pending activation detection result indicates that the environment image includes at least one pending activation target, i.e., the pending activation detection result includes at least one pending activation detection box. The preset usage intent corresponds to the "Follow Me" usage intent. If both the pending activation target can be detected in the direction of the sound source (the pending activation direction) and the collected voice data includes the "Follow Me" intent, the target identifier is determined based on the pending activation identifier corresponding to the pending activation target, and the target following process begins.

[0137] Among them, determining the target identifier based on the to-be-started identifier corresponding to the to-be-started target includes: first determining the feature data corresponding to the to-be-started target based on the to-be-started detection frame corresponding to the to-be-started target; then performing matching processing from a preset sample feature library to obtain the sample identifier corresponding to the feature data, and obtain the to-be-started identifier, wherein, if no match is found, the feature data is stored in the sample feature library, and at the same time, a corresponding to-be-started identifier is generated for the feature data, and the to-be-started identifier is stored in the sample feature library as a sample identifier, so as to facilitate feature matching processing in the target following process.

[0138] In one possible implementation, the to-be-started detection result includes multiple to-be-started targets. In the process of determining the target identifier based on the to-be-started identifier corresponding to the to-be-started target, the to-be-started detection frames corresponding to each to-be-started target are first compared in size, and the to-be-started target with the largest to-be-started detection frame among each to-be-started target is selected as the follow-up target, and the target identifier corresponding to the follow-up target is obtained.

[0139] The method provided by the above embodiment, in the process of the following device determining the following target, if voice data including the target wake-up word is detected, the voice data is processed for sound source positioning to determine the direction to be started; the voice data is processed for voice recognition to obtain the usage intention corresponding to the voice data; the environmental image corresponding to the direction to be started is obtained, and the environmental image is processed for target detection to obtain the detection result to be started; if the detection result to be started indicates that the environmental image includes at least one target to be started, and the usage intention is a preset following intention, the target identifier is determined based on the to-be-started identifier corresponding to the target to be started. In this way, the process of selecting and switching the following target using voice data between the following equipment and the following target is realized, making human-computer interaction more natural, improving the following experience, and increasing the flexibility of following.

[0140] In one embodiment, based on the embodiment shown in FIG1 , please refer to FIG6 , which provides a process for obtaining the second coordinates of the tracking target based on the target detection frame and the first coordinates corresponding to the tracking device. As shown in FIG6 , the process includes:

[0141] Step 602: Perform depth estimation based on the target detection frame and the preset calibration frame to obtain the target distance between the tracking target and the tracking device.

[0142] For example, the preset calibration frame corresponds to a human target of average height (1.7m). According to the size of the target detection frame on the target image and the size of the calibration frame on the target image, depth estimation processing is performed to obtain the target distance between the tracking target and the tracking device.

[0143] Step 604 : Calculate the deflection angle based on the target detection frame and the image center point of the target image to obtain a target angle between the tracking target and the tracking device.

[0144] Exemplarily, according to the horizontal distance and the vertical distance between the target detection frame and the image center point of the target image, the deflection angle of the follow-up target relative to the follow-up device is obtained to obtain the target angle.

[0145] Step 606 , performing calculation based on the target distance, the target angle, and the first coordinate to obtain a second coordinate.

[0146] The distance of the follow target relative to the follow device is determined based on the target distance and the target angle, and the second coordinate of the follow target in the world coordinate system is obtained according to the first coordinate of the follow device.

[0147] In one possible implementation, the estimated world coordinates corresponding to the tracking target are filtered to obtain the second coordinates to prevent excessive fluctuations in the world coordinates corresponding to the tracking target due to unexpected situations, thereby improving the reliability of the second coordinates. Exemplarily, the filtering process may adopt a low-pass filtering method.

[0148] In another possible implementation, the estimated world coordinates corresponding to the tracking target are directly used as the second coordinates.

[0149] In the target following method provided by this embodiment, the target distance between the following target and the following device is obtained by performing depth estimation processing based on the target detection frame and the preset calibration frame, and the target angle between the following target and the following device is obtained by performing deflection angle calculation processing based on the target detection frame and the image center point of the current image; the second coordinate is obtained by performing calculation based on the target distance, the target angle and the first coordinate. In this way, the target distance is obtained using the preset calibration frame, and the target angle is obtained based on the target detection frame and the image center point. Then, based on the first coordinate of the following device, a simple calculation is performed to obtain the second coordinate of the following target in real time. The calculation complexity is low, the calculation efficiency is high, and the consumption of computing resources can be reduced.

[0150] In one embodiment, based on the embodiment shown in FIG1 , in this embodiment, the historical navigation data includes the historical navigation path and the historical destination coordinates, and the target navigation data includes the target navigation path and the target destination coordinates. As shown in FIG7 , the process provided in this embodiment for updating the historical navigation data of the following device according to the second coordinate to obtain the target navigation data includes:

[0151] Step 702: Update the historical navigation path using the second coordinate as a new path point to obtain a target navigation path.

[0152] Step 704 : Calculate the distance between the second coordinate and the historical target coordinate to obtain a distance difference.

[0153] Step 706: If the distance difference is greater than the difference threshold, determine the target destination coordinates based on the second coordinates.

[0154] Step 708: If the distance difference is less than or equal to the difference threshold, the historical destination coordinates are determined as the target destination coordinates.

[0155] In one embodiment, based on the embodiment shown in FIG7 , this embodiment provides a process for controlling the movement of a follower device based on target navigation data. As shown in FIG8 , the process includes:

[0156] Step 802: Generate movement instructions corresponding to the following device according to the target navigation data.

[0157] Exemplarily, the movement instruction is used to control the following device to move along the target navigation path toward a position corresponding to the target destination coordinates.

[0158] Step 804 : Perform local path planning based on the environmental point cloud data acquired by the following device to obtain an avoidance instruction corresponding to the following device.

[0159] Among them, environmental point cloud data can be obtained through the lidar sensor installed on the following device.

[0160] An exemplary lidar sensor includes an internal rotation mechanism and a single-line lidar. The lidar emits laser beams and receives reflected laser beams, acquiring and recording the distance and intensity of each laser beam. The internal rotation mechanism rotates the laser beams horizontally, enabling 360-degree scanning and ranging of the surrounding environment. The lidar converts the received distance information into environmental point cloud data and, through communication with the control system in the follower device, transmits the point cloud data to the control system for subsequent processing.

[0161] Local path planning is performed based on environmental point cloud data, generating avoidance instructions to achieve real-time obstacle avoidance. For example, algorithms such as the TEB (Time Elastic Band) algorithm, the artificial potential field method, and the DWA (Dynamic Window Approach) algorithm can be used for local path planning.

[0162] Step 806: Control the movement of the following device based on the movement instruction and the avoidance instruction.

[0163] The target following method provided in this embodiment, in addition to generating movement instructions corresponding to the following device based on the target navigation path and the target destination coordinates, also performs local path planning based on the point cloud data obtained by the following device to obtain avoidance instructions corresponding to the following device; controlling the movement of the following device based on the movement instructions and avoidance instructions can enable the following device to achieve real-time obstacle avoidance during the process of following the target, thereby improving the safety of the target following task and further enhancing the flexibility of the following device in performing the target following task.

[0164] In one embodiment, the provided target following method further includes: generating a target lost instruction if the following target cannot be detected for a continuous preset time period.

[0165] The fact that the target navigation path has not been updated for a preset period of time may be used, or the fact that the target detection frame has not been obtained for a preset period of time may be used to determine that the target cannot be detected for a preset period of time.

[0166] In one possible implementation, when obtaining the target navigation route, the route update time is determined based on the current device time. If the time difference between the current time of the following device and the route update time is greater than a preset time threshold, a target lost instruction is generated.

[0167] The path update time is a variable value that is updated continuously as the navigation path is updated. When the following device cannot detect the target detection frame corresponding to the target, that is, when the second coordinate of the target cannot be updated, and therefore when the navigation path cannot be updated, the device determines whether the target is lost by determining the duration of the navigation path stoppage, that is, whether the time difference between the current time of the following device and the path update time is greater than a preset time difference. When the time difference between the current time of the following device and the path update time is greater than a preset time threshold, that is, the navigation path has not been updated for a period of time, the target is considered lost and a target loss instruction is generated. The target loss instruction is used to control the following device to perform the corresponding action.

[0168] In another possible implementation, when obtaining the target detection frame corresponding to the follow target, the target detection time is determined based on the current device time; if the time difference between the current time of the follow device and the target detection time is greater than a preset time threshold, a target lost instruction is generated.

[0169] Exemplarily, the target lost instruction is used to control the following device to stop following, emit a sound to remind the following target, and then enter a standby state, waiting to be awakened by the following target.

[0170] As another example, the target lost instruction is used to control the following device to enter a search state, and search for the following target according to a preset method along a certain route, a certain direction, or a certain area.

[0171] In one embodiment, referring to FIG9 , the target following method provided further includes:

[0172] Step 902: In response to the target lost instruction, determine the search center coordinates based on the second coordinates.

[0173] The second coordinate refers to the second coordinate most recently determined based on the target detection frame. If filtering is used in the process of obtaining the second coordinate, the world coordinate before filtering is determined as the search center coordinate. If filtering is not performed in the process of obtaining the second coordinate, the second coordinate is directly determined as the search center coordinate.

[0174] This embodiment enters a search state after the tracking target is lost, and uses the position of the tracking target obtained by the most recent detection as the search center.

[0175] Step 904: Determine an area of ​​a preset size as a first search area according to the search center coordinates.

[0176] Exemplarily, a 4m×4m area around the search center coordinates is used as the first search area, that is, the search is performed in a 4m×4m local map around the search center coordinates.

[0177] Step 906 , controlling the following device to traverse the first search area, and continuously performing feature matching processing on the acquired search image until a first matching result indicates that a following target exists in the first search area, or the traversal ends.

[0178] Exemplarily, the traversal search process of step 906 includes:

[0179] Step A1: Perform feature matching processing on the search image acquired by the following device at the current position to obtain a first matching result.

[0180] It should be noted that if filtering is used in the process of obtaining the second coordinates, the following device must first be controlled to move to the position corresponding to the search center coordinates, and then the feature matching process is performed on the search image obtained at the current position.

[0181] The search image refers to an image captured by the following device in a search state. The first matching result is used to indicate whether a following target exists in the first search area. For example, whether a following target exists in the first search area can be indicated by whether the corresponding search image includes a target identifier of the following target.

[0182] Step A2: If the first matching result indicates that there is no tracking target in the first search area, the area corresponding to the search image in the first search area is set as a known area, and the area other than the known area in the first search area is set as an unknown area.

[0183] Step A3: Control the following device to move to the position corresponding to the coordinates of the boundary point, and repeatedly perform feature matching processing on the search image obtained by the following device at the current position to obtain a first matching result; if the first matching result indicates that there is no following target in the first search area, then the area corresponding to the search image in the first search area is set as a known area, and the area in the first search area other than the known area is set as an unknown area. This process is repeated until the first matching result indicates that there is a following target in the first search area, or there is no available boundary in the first search area (that is, the traversal is completed).

[0184] After the traversal search begins, the visible range of the image acquisition device of the following device (illustratively, a 60° field of view and a distance of 3 meters) is updated to the known area; if no matching tracking target is found within the current visible range, the following device continues to move to the boundary between the known and unknown areas until there is no longer a usable boundary in the first search area. Exemplarily, the known and unknown areas are continuously updated as the following device moves, while the boundary between the unknown and known areas is updated and determined by a breadth-first search based on the known and unknown areas within the first search area at the time the following device reaches the boundary coordinates.

[0185] Exemplarily, the boundary point coordinates are generally determined based on the middle point of the boundary, and the direction is: back to the known area and toward the unknown area. For example, when the middle point coordinates of the boundary are determined as the destination position of the following device, it is determined whether the following device will collide with an obstacle when moving to the location corresponding to the middle point coordinates; if the following device will not collide with an obstacle when moving to the location corresponding to the middle point coordinates, the middle point coordinates of the boundary are determined as the boundary point coordinates. If the following device will collide with an obstacle when moving to the location corresponding to the middle point coordinates, a point is randomly selected within a preset range around the middle point coordinates (exemplarily, 0.5 meters × 0.5 meters, 1 arc toward the threshold) as the new destination position; it is determined whether the following device will collide with an obstacle when moving to the new destination position. If the following device will collide with an obstacle when moving to the new destination position, it is considered that the random point selection has failed, the exploration of the boundary is abandoned, and the boundary is marked as unavailable; if the following device will not collide with an obstacle when moving to the new destination position, it is considered that the new destination position is used as the boundary point coordinates.

[0186] Step 908: If the first matching result indicates that there is a follow-up target in the first search area, obtain a target detection frame corresponding to the follow-up target.

[0187] Among them, when there is a following target in the first search area, the target detection frame corresponding to the following target is obtained, and then the second coordinate of the following target is obtained based on the target detection frame, that is, the first coordinate corresponding to the following device. The historical navigation data of the following device is updated according to the second coordinate to obtain the target navigation data, and the movement of the following device is controlled based on the target navigation data.

[0188] Optionally, if all the first search areas are set as known areas, it is considered that no follow target is found in the first search area, indicating that the follow target is completely lost. At this time, the follow device can be controlled to emit a sound to remind the follow target, and then enter standby mode, waiting for the follow target to wake up.

[0189] The target following method provided in this embodiment incorporates an automatic search method when the target is lost. The following device can proactively search areas where the target may appear, thereby ensuring the success rate of following, thereby improving the robustness of the target following method and enhancing the user experience.

[0190] In one embodiment, referring to FIG10 , the target following method further includes:

[0191] Step 1002: During the execution of the target lost instruction, if voice data including the target wake-up word is detected, the voice data is processed for sound source localization to determine a second search direction.

[0192] In this embodiment, if the target finds that the following device is already tracking it and the search direction is moving away from it, it can wake up the following device by using a wake-up word. After hearing the wake-up word, the following device locates the sound source and determines a second search direction.

[0193] Step 1004: Control the following device to turn to a second search direction, and determine a second search area based on the second search direction and the first coordinates.

[0194] Exemplarily, the following device may define an area of ​​a preset size as the second search area along the second search direction based on the first coordinate.

[0195] Step 1006 , controlling the following device to traverse the second search area, and continuously performing feature matching processing on the acquired search image until a second matching result indicates that a following target exists in the second search area, or the traversal ends.

[0196] Exemplarily, the traversal process shown in step 1006 may include:

[0197] Step B1: Perform feature matching processing on the search image acquired by the following device at the current position to obtain a second matching result.

[0198] Step B2: if the second matching result indicates that there is no tracking target in the second search area, then the area corresponding to the search image in the second search area is set as a known area, and the area other than the known area in the second search area is set as an unknown area;

[0199] Step B3: Control the following device to move to the position corresponding to the coordinates of the boundary point, and repeatedly perform feature matching processing on the search image obtained by the following device at the current position to obtain a second matching result; if the second matching result indicates that there is no following target in the second search area, then the area corresponding to the search image in the second search area is set as a known area, and the area in the second search area other than the known area is set as an unknown area. This process is repeated until the second matching result indicates that there is a following target in the second search area, or there is no available boundary in the second search area (that is, the traversal is completed).

[0200] The method for determining the boundary point coordinates and the boundary can refer to the description in step 906 in the above embodiment, and will not be repeated here.

[0201] Step 1008: If the second matching result indicates that there is a follow-up target in the second search area, obtain a target detection frame corresponding to the follow-up target.

[0202] The target following method provided in this embodiment can remind the following device of the following direction through voice when the following device enters the automatic search state, thereby improving the flexibility of the target following method and enhancing the user experience.

[0203] In one embodiment, a target following method is provided, comprising:

[0204] Step C1: visually initiate following, and / or, wake-up word initiate following.

[0205] Optionally, visually initiated following includes: continuously performing feature matching processing on the environmental image obtained by the following device to obtain a to-be-started identifier; for the to-be-started identifier, detecting whether the to-be-started identifier is obtained continuously for a preset time period; if the to-be-started identifier is obtained continuously for a preset time period, obtaining the face angle corresponding to the to-be-started identifier; if the face angle belongs to the preset angle range, using the to-be-started identifier as the target identifier.

[0206] During the process of visually initiated following, optionally, feature matching processing is continuously performed on the environmental image acquired by the following device to obtain a to-be-started identification, including: continuously performing feature extraction processing on the environmental image acquired by the following device to obtain the to-be-started feature; using the sample identification corresponding to the sample feature that successfully matches the to-be-started feature among multiple sample features in the sample feature library as the to-be-started identification; if the to-be-started feature fails to match the sample features in the sample feature library, generating a to-be-started identification for the to-be-started feature, and using the to-be-started identification as the sample identification and storing the to-be-started feature as the sample feature in the sample feature library.

[0207] Optionally, the wake-up word start-up follow-up includes: if voice data including the target wake-up word is detected, performing sound source localization processing on the voice data to determine the direction to be started; performing voice recognition processing on the voice data to obtain the usage intention corresponding to the voice data; obtaining the environmental image corresponding to the direction to be started, and performing target detection processing on the environmental image to obtain the detection result to be started; if the detection result to be started indicates that the environmental image includes at least one target to be started, and the usage intention is a preset follow-up intention, then determining the target identifier based on the to-be-started identifier corresponding to the target to be started.

[0208] Step C2, real-time estimation of the human body position, includes: performing target detection processing and feature matching processing on the acquired target image to obtain a target detection frame corresponding to the follow target; obtaining the second coordinates of the follow target based on the target detection frame and the first coordinates corresponding to the follow device; and updating the historical navigation data of the follow device based on the second coordinates to obtain target navigation data.

[0209] Among them, optionally, target detection processing and feature matching processing are performed on the acquired target image to obtain a target detection frame corresponding to the follow target, including: performing target detection processing on the target image to obtain a candidate detection frame and candidate features corresponding to the candidate target; based on a sample feature library including sample features and sample identifiers corresponding to the sample features, matching processing is performed on the candidate features to obtain candidate identifiers corresponding to the candidate features; if the candidate identifier is the same as the target identifier of the follow target, the candidate detection frame corresponding to the candidate target is used as the target detection frame.

[0210] Optionally, if all candidate identifiers are different from the target identifier of the follow target, the target detection frame corresponding to the follow target cannot be obtained.

[0211] Among them, optionally, the second coordinate of the following target is obtained according to the target detection frame and the first coordinate corresponding to the following device, including: performing depth estimation processing based on the target detection frame and the preset calibration frame to obtain the target distance between the following target and the following device; performing deflection angle calculation processing based on the target detection frame and the image center point of the target image to obtain the target angle between the following target and the following device; and performing calculation based on the target distance, target angle and the first coordinate to obtain the second coordinate.

[0212] Among them, optionally, the historical navigation data includes historical navigation paths and historical destination coordinates, and the target navigation data includes target navigation paths and target destination coordinates; updating the historical navigation data of the following device according to the second coordinates to obtain target navigation data includes: updating the historical navigation path using the second coordinates as a new path point to obtain the target navigation path; calculating the distance between the second coordinates and the historical target coordinates to obtain a distance difference; if the distance difference is greater than a difference threshold, determining the target destination coordinates based on the second coordinates; if the distance difference is less than or equal to the difference threshold, determining the historical destination coordinates as the target destination coordinates.

[0213] Step C3, following movement, includes: generating movement instructions corresponding to the following device according to the target navigation path and the target destination coordinates; performing local path planning based on the environmental point cloud data obtained by the following device to obtain avoidance instructions corresponding to the following device; and controlling the movement of the following device based on the movement instructions and avoidance instructions.

[0214] Step C4: If the target cannot be detected for a preset time period, a target lost instruction is generated.

[0215] Step C5, in response to the target lost instruction, performs an automatic search, including: determining the search center coordinates based on the second coordinates; determining an area of ​​a preset size as a first search area according to the search center coordinates; controlling the following device to traverse the first search area in a breadth-first search manner, and continuously performing a feature matching process on the acquired search image until a first matching result indicates that a following target exists in the first search area, or the traversal ends; if the first matching result indicates that a following target exists in the first search area, obtaining a target detection frame corresponding to the following target.

[0216] Step C6, in the process of executing the instruction in response to the target loss, automatic search can be performed with the assistance of wake-up words, including: in the process of executing the instruction in response to the target loss, if voice data including the target wake-up word is detected, the voice data is processed for sound source positioning to determine the second search direction; the following device is controlled to turn to the second search direction, and based on the second search direction and the first coordinate, it is determined as the second search area; the following device is controlled to traverse the second search area in a breadth-first search manner, and the process of performing feature matching processing on the acquired search image is continuously performed until the feature matching result indicates that there is a follow target in the second search area, or the traversal is completed; if the second matching result indicates that there is a follow target in the second search area, the target detection box corresponding to the follow target is obtained.

[0217] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0218] Based on the same inventive concept, embodiments of the present application also provide a target following device for implementing the target following method described above. The solution provided by this device is similar to the solution described in the method described above. Therefore, the specific limitations of one or more target following device embodiments provided below can be found in the limitations of the target following method described above and will not be repeated here.

[0219] In one embodiment, as shown in FIG11 , a target following apparatus is provided for a movable following device, comprising: a target detection module 1102 , a coordinate determination module 1104 , and a navigation control module 1106 , wherein:

[0220] The target detection module 1102 is used to perform target detection processing and feature matching processing on the acquired target image to obtain a target detection frame corresponding to the follow target;

[0221] A coordinate determination module 1104 is configured to obtain a second coordinate of the tracking target based on the target detection frame and the first coordinate corresponding to the tracking device;

[0222] The navigation control module 1106 is configured to update the historical navigation data of the following device according to the second coordinates, obtain target navigation data, and control the movement of the following device based on the target navigation data.

[0223] In one embodiment, the target detection module 1102 is used to perform target detection processing on the target image to obtain candidate detection frames and candidate features corresponding to the candidate targets, and perform matching processing on the candidate features based on a sample feature library including sample features and sample identifiers corresponding to the sample features to obtain candidate identifiers corresponding to the candidate features. If the candidate identifier is the same as the target identifier of the followed target, the candidate detection frame corresponding to the candidate target is used as the target detection frame.

[0224] In one embodiment, the target following device also includes a following start module, wherein the following start module is used to continuously perform feature matching processing on the environmental image obtained by the following device to obtain a to-be-started identifier. For the to-be-started identifier, it is detected whether the to-be-started identifier is obtained continuously for a preset time period. If the to-be-started identifier is obtained continuously for a preset time period, the face angle corresponding to the to-be-started identifier is obtained. If the face angle belongs to the preset angle range, the to-be-started identifier is used as the target identifier.

[0225] In one embodiment, the target following device also includes a following start module, wherein the following start module is used to perform sound source localization processing on the voice data if voice data including the target wake-up word is detected, determine the direction to be started, perform voice recognition processing on the voice data, obtain the usage intention corresponding to the voice data, obtain the environmental image corresponding to the direction to be started, and perform target detection processing on the environmental image to obtain a detection result to be started. If the detection result to be started indicates that the environmental image includes at least one target to be started, and the usage intention is a preset following intention, the target identifier is determined based on the to-be-started identifier corresponding to the target to be started.

[0226] In one embodiment, the follow-up start module is specifically used to continuously perform feature extraction processing on the environmental image obtained by the following device to obtain the feature to be started; the sample identifier corresponding to the sample feature that successfully matches the feature to be started among the multiple sample features in the sample feature library is used as the identifier to be started; if the feature to be started fails to match the sample features in the sample feature library, a identifier to be started is generated for the feature to be started, and the identifier to be started is used as the sample identifier and the feature to be started is stored in the sample feature library as the sample feature.

[0227] In one embodiment, the coordinate determination module 1104 is specifically used to perform depth estimation processing based on the target detection frame and the preset calibration frame to obtain the target distance between the follow target and the follow device, perform deflection angle calculation processing based on the target detection frame and the image center point of the target image to obtain the target angle between the follow target and the follow device, and perform calculation based on the target distance, target angle and the first coordinate to obtain the second coordinate.

[0228] In one embodiment, the historical navigation data includes the historical navigation path and the historical destination coordinates, and the target navigation data includes the target navigation path and the target destination coordinates; the navigation control module 1106 is specifically used to update the historical navigation path using the second coordinate as a new path point to obtain the target navigation path, calculate the distance between the second coordinate and the historical target coordinates, and obtain the distance difference. If the distance difference is greater than the difference threshold, the target destination coordinates are determined based on the second coordinates; if the distance difference is less than or equal to the difference threshold, the historical destination coordinates are determined as the target destination coordinates.

[0229] In one embodiment, the navigation control module 1106 is specifically used to generate movement instructions corresponding to the following device based on the target navigation data; perform local path planning based on the environmental point cloud data obtained by the following device to obtain avoidance instructions corresponding to the following device; and control the movement of the following device based on the movement instructions and avoidance instructions.

[0230] In one embodiment, the target following device further comprises a lost search module, wherein the lost search module is configured to generate a target lost instruction if the following target cannot be detected for a continuous preset time period.

[0231] In one embodiment, the lost search module is also used to respond to a target lost instruction, determine the search center coordinates based on the second coordinates, determine an area of ​​a preset size as a first search area according to the search center coordinates, control the following device to traverse the first search area in a breadth-first search manner, and continuously perform feature matching processing on the acquired search image until the first matching result indicates that there is a following target in the first search area, or, the traversal ends, if the first matching result indicates that there is a following target in the first search area, then obtain the target detection box corresponding to the following target.

[0232] In one embodiment, the lost search module is also used to, in the process of executing a response to a target lost instruction, if voice data including a target wake-up word is detected, perform sound source localization processing on the voice data, determine a second search direction, control the following device to turn to the second search direction, and determine the second search area based on the second search direction and the first coordinate, control the following device to traverse the second search area in a breadth-first search manner, and continue to perform feature matching processing on the acquired search image until the feature matching result indicates that there is a follow target in the second search area, or, the traversal ends, if the second matching result indicates that there is a follow target in the second search area, obtain the target detection box corresponding to the follow target.

[0233] Each module in the target tracking device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0234] In one embodiment, a movable following device is provided, which includes an image acquisition device, a voice acquisition device, a laser radar sensor, and a target following device, wherein the target following device is used to implement each step of the target following method provided in the above embodiment.

[0235] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in Figure 12. The computer device includes a processor, a memory, an input / output interface (I / O), and a communication interface. The processor, memory, and I / O interface are connected via a system bus, and the communication interface is connected to the system bus via the I / O interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store data called when executing a target following method. The I / O interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a target following method is implemented.

[0236] Those skilled in the art will understand that the structure shown in FIG12 is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0237] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0238] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0239] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0240] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0241] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0242] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0243] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A target following method, characterized in that: For a movable following device, the method comprises: Performing target detection processing and feature matching processing on the acquired target image to obtain a target detection frame corresponding to the follow target; Obtaining a second coordinate of the following target according to the target detection frame and the first coordinate corresponding to the following device; The historical navigation data of the following device is updated according to the second coordinates to obtain target navigation data, and the movement of the following device is controlled based on the target navigation data.

2. The method according to claim 1, characterized in that The performing target detection processing and feature matching processing on the acquired target image to obtain a target detection frame corresponding to the follow target includes: Performing target detection processing on the target image to obtain candidate detection frames and candidate features corresponding to the candidate targets; Based on a sample feature library including sample features and sample identifiers corresponding to the sample features, matching processing is performed on the candidate features to obtain candidate identifiers corresponding to the candidate features; If the candidate identifier is the same as the target identifier of the follow target, the candidate detection frame corresponding to the candidate target is used as the target detection frame.

3. The method according to claim 2, characterized in that Before performing target detection processing and feature matching processing on the acquired target image to obtain a target detection frame corresponding to the follow target, the method further includes: Continuously performing feature matching processing on the environment image acquired by the following device to obtain a to-be-started mark; For the to-be-started mark, detecting whether the to-be-started mark is obtained continuously for a preset time period; If the to-be-started mark is obtained continuously for a preset time period, the face angle corresponding to the to-be-started mark is obtained; If the face angle belongs to the preset angle range, the to-be-started marker is used as the target marker.

4. The method according to claim 2, characterized in that: Before performing target detection processing and feature matching processing on the acquired target image to obtain a target detection frame corresponding to the follow target, the method further includes: If voice data including the target wake-up word is detected, the voice data is processed for sound source localization to determine the direction to be activated; Performing speech recognition processing on the speech data to obtain the usage intention corresponding to the speech data; Acquire an environment image corresponding to the direction to be started, and perform target detection processing on the environment image to obtain a detection result to be started; If the to-be-started detection result indicates that the environment image includes at least one to-be-started target, and the use intention is a preset follow intention, the target identifier is determined based on the to-be-started identifier corresponding to the to-be-started target.

5. The method according to claim 3, characterized in that: The continuously performing feature matching processing on the environment image acquired by the following device to obtain the to-be-started mark includes: Continuously performing feature extraction processing on the environment image acquired by the following device to obtain features to be started; Using a sample identifier corresponding to a sample feature that successfully matches the feature to be started among multiple sample features in the sample feature library as the identifier to be started; If the feature to be activated fails to match the sample feature in the sample feature library, the identifier to be activated is generated for the feature to be activated, and the identifier to be activated is used as a sample identifier and the feature to be activated is stored in the sample feature library as a sample feature.

6. The method according to claim 1, characterized in that The obtaining, according to the target detection frame and the first coordinates corresponding to the following device, the second coordinates of the following target includes: Performing depth estimation processing according to the target detection frame and the preset calibration frame to obtain the target distance between the following target and the following device; Performing a deflection angle calculation process based on the target detection frame and the image center point of the target image to obtain a target angle between the following target and the following device; The second coordinate is obtained by performing calculation according to the target distance, the target angle and the first coordinate.

7. The method according to claim 1, characterized in that The historical navigation data includes a historical navigation path and a historical destination coordinate, and the target navigation data includes a target navigation path and a target destination coordinate; and the updating of the historical navigation data of the following device according to the second coordinate to obtain the target navigation data includes: Using the second coordinate as a new path point to update the historical navigation path, to obtain the target navigation path; Calculating the distance between the second coordinate and the historical target coordinate to obtain a distance difference; If the distance difference is greater than a difference threshold, determining the target destination coordinates based on the second coordinates; If the distance difference is less than or equal to the difference threshold, the historical destination coordinates are determined as the target destination coordinates.

8. The method according to claim 7, characterized in that The controlling the movement of the following device based on the target navigation data comprises: Generating a movement instruction corresponding to the following device according to the target navigation data; Performing local path planning according to the environmental point cloud data acquired by the following device to obtain an avoidance instruction corresponding to the following device; The following device is controlled to move based on the movement instruction and the avoidance instruction.

9. The method according to claim 1, characterized in that: The method further comprises: If the following target cannot be detected for a preset time period, a target lost instruction is generated.

10. The method according to claim 9, characterized in that The method further comprises: In response to the target lost instruction, determining a search center coordinate based on the second coordinate; Determining an area of ​​a preset size as a first search area according to the search center coordinates; Controlling the following device to traverse the first search area, and continuously performing a process of performing feature matching processing on the acquired search image, until a first matching result indicates that the following target exists in the first search area, or the traversal ends; If the first matching result indicates that the follow target exists in the first search area, an object detection frame corresponding to the follow target is obtained.

11. The method according to claim 10, characterized in that The method further comprises: In the process of executing the instruction in response to the target loss, if voice data including the target wake-up word is detected, the voice data is processed for sound source localization to determine a second search direction; Control the following device to turn to the second search direction, and determine the second search area based on the second search direction and the first coordinates; Controlling the following device to traverse the second search area, and continuously performing a process of performing feature matching processing on the acquired search image, until a second matching result indicates that the following target exists in the second search area, or the traversal ends; If the second matching result indicates that the follow target exists in the second search area, an object detection frame corresponding to the follow target is obtained.

12. A target following device, characterized in that: For a movable following device, the device comprises: The target detection module is used to perform target detection processing and feature matching processing on the acquired target image to obtain a target detection frame corresponding to the follow target; A coordinate determination module, used for obtaining a second coordinate of the follow target according to the target detection frame and the first coordinate corresponding to the follow device; The navigation control module is used to update the historical navigation data of the following device according to the second coordinates to obtain target navigation data, and control the movement of the following device based on the target navigation data.

13. A movable following device, characterized in that: The device includes an image acquisition device, a voice acquisition device, a laser radar sensor and a target following device; The target following device is used to implement the steps of any one of the methods of claims 1 to 11.

14. A movable following device, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 11 are implemented.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images