Target following method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a target following method and device, computer equipment, a readable storage medium and a program product. The method comprises the following steps: obtaining a part detection frame of a moving object in a target scene in a current frame image at a current moment, wherein the part detection frame is used for marking a target part of the moving object; projecting a point cloud collected by a radar sensor to the current frame image to obtain a projection point set corresponding to the part detection frame; identifying a target moving object from the moving objects; determining a target projection point set corresponding to the target moving object from the projection point set based on a part detection frame corresponding to the target moving object; obtaining object information of the target moving object at the current moment based on the target projection point set; and obtaining following information of the mobile robot based on the object information so as to follow the target moving object based on the following information. By adopting the method, the target following stability of the robot can be improved, and the probability of missing or wrong target following is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robots, in particular to a target following method and device, computer equipment, computer readable storage medium and computer program product. BACKGROUND

[0002] Robots are a concentrated embodiment of automation technology, sensor technology and artificial intelligence technology. With the development of these technologies, there is a demand for robots to follow operators, i.e. robots detect the position of an operator (to-be-followed object) and generate a movement path for the robot based on the position to achieve the purpose of target following.

[0003] In traditional technology, the autonomous following of robots is usually achieved by the operator holding an ultrasonic, ultra-wideband (UWB) or other emission module in cooperation with a pre-installed receiving module on the robot. This solution not only requires additional material costs, but also unstable following, often losing the to-be-followed object. SUMMARY

[0004] Therefore, it is necessary to provide a target following method, device, computer equipment, computer readable storage medium and computer program product capable of improving the stability of robot following targets and reducing the probability of losing or following the wrong target.

[0005] In a first aspect, the present application provides a target following method, comprising:

[0006] obtaining a part detection box of a moving object in a target scene in a current frame image at a current time, the part detection box being used to label a target part of the moving object;

[0007] projecting point clouds collected by a radar sensor to the current frame image to obtain a projection point set corresponding to the part detection box;

[0008] identifying a target moving object from the moving object;

[0009] determining a target projection point set corresponding to the target moving object from the projection point set based on the part detection box corresponding to the target moving object;

[0010] obtaining object information of the target moving object at the current time based on the target projection point set;

[0011] obtaining following information of a mobile robot based on the object information, and following the target moving object based on the following information.

[0012] In a second aspect, the present application further provides a target following device, comprising:

[0013] An image detection module is configured to obtain a part detection frame of a mobile object in a current frame image of a target scene at a current time, and the part detection frame is used to mark a target part of the mobile object.

[0014] A radar detection module is configured to project a point cloud collected by a radar sensor to the current frame image to obtain a projection point set corresponding to the part detection frame.

[0015] An identification module is configured to identify a target mobile object from the mobile object.

[0016] An association module is configured to determine a target projection point set corresponding to the target mobile object from the projection point set based on a part detection frame corresponding to the target mobile object.

[0017] A determination module is configured to obtain object information of the target mobile object at the current time based on the target projection point set.

[0018] A following module is configured to obtain following information of the mobile robot based on the object information, and to follow the target mobile object based on the following information.

[0019] In a third aspect, a computer device is provided, which includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the steps described in the above method are implemented.

[0020] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the steps described in the above method are implemented.

[0021] In a fifth aspect, a computer program product is provided, which includes a computer program. When the computer program is executed by a processor, the steps described in the above method are implemented.

[0022] The target following method, device, computer device, computer readable storage medium and computer program product can obtain a current frame image at a current time through a visual sensor built in the mobile robot, obtain a part detection box of a moving object in the target scene in the current frame image, and obtain a point cloud corresponding to the target scene at the current time through a radar sensor built in the mobile robot. Each point of the point cloud is projected to the current frame image to obtain a projection point set corresponding to the part detection box. In this way, the projection point set corresponding to the moving object in the target scene can be screened out. Then, a target moving object is identified from the moving object. A target projection point set corresponding to the target moving object is determined from the projection point set based on the part detection box of the target moving object. In this way, the object information of the target moving object at the current time can be obtained based on the target projection point set corresponding to the target moving object. The following information of the mobile robot can be obtained based on the object information. Through the above steps, the moving related information of the following target can be obtained based on the originally built-in sensor of the mobile robot to save cost. The object information, such as the position information, of the following target can be obtained through the comprehensive use of multiple sensors. In this way, the accuracy of identifying the following target can be improved, the probability of losing and misidentifying the following target can be reduced, and the stability of target following can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the description of the embodiments of the present application or the related art will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other related drawings can also be obtained without creative labor.

[0024] Figure 1 An application environment diagram of the target following method in an embodiment;

[0025] Figure 2 A flowchart of the target following method in an embodiment;

[0026] Figure 3 A schematic diagram of the detection box and the target projection point set in an embodiment;

[0027] Figure 4 A flowchart of the step of determining the target projection point set in an embodiment;

[0028] Figure 5 A schematic diagram of the candidate projection point set in an embodiment;

[0029] Figure 6 A flowchart of the step of determining the target projection point set in another embodiment;

[0030] Figure 7 a schematic diagram of bounding boxes identified in images collected at different times in one embodiment;

[0031] Figure 8 a schematic diagram of a process for determining a target moving object in one embodiment;

[0032] Figure 9 a schematic diagram of a process for obtaining following information in one embodiment;

[0033] Figure 10 a schematic diagram of a process for determining a following target in one embodiment;

[0034] Figure 11 a schematic diagram of a process for determining information of a following target in one embodiment;

[0035] Figure 12 a schematic diagram of a process for obtaining following information of a mobile robot in another embodiment;

[0036] Figure 13 a block diagram of a structure of a target following device in one embodiment;

[0037] Figure 14 a block diagram of an internal structure of a computer device in one embodiment;

[0038] Figure 15 a block diagram of an internal structure of a computer device in another embodiment. DETAILED DESCRIPTION

[0039] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not used to limit the present application.

[0040] The target following method provided by the embodiments of the present application can be applied to, for example, Figure 1The application environment shown in the mobile robot 104 follows the moving object 102. Among them, the mobile robot 104 can be various self-moving devices, such as cleaning robots, food delivery robots, object carrying robots, and other service robots, which are usually configured with various sensors for navigation and obstacle avoidance, such as visual sensors such as cameras, cameras, and ranging sensors such as laser radars. It can be understood that the mobile robot 104 can cooperate with various computers, smartphones, tablets, Internet of Things devices, or portable wearable devices to achieve the target following method provided in the embodiments of the present application. The mobile robot 104 can also cooperate with the server to achieve the target following method provided in the embodiments of the present application, wherein the server can be an independent physical server, or a service node in a blockchain system, and the service nodes in the blockchain system form a peer-to-peer (P2P, Peer To Peer) network, and the P2P protocol is an application layer protocol running on the transmission control protocol (TCP, Transmission Control Protocol) protocol. In addition, the server can also be a server cluster composed of multiple physical servers, which can be a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms. Basic cloud computing services such as platform. The mobile robot 104 and the terminal or server can be connected and communicated through Bluetooth, USB (Universal Serial Bus, Universal Serial Bus) or network, etc., which is not limited in the present application.

[0041] In an exemplary embodiment, as shown in Figure 2 , a target following method is provided, which is applied to the mobile robot (such as a carrying mobile robot) in Figure 1 , including the following steps S202 to S212. Among them:

[0042] Step S202, obtaining the part detection frame of the moving object in the target scene in the current frame image at the current time.

[0043] Exemplarily, in order to follow the target at the current time, the current frame image obtained by the visual sensor shooting the target scene at the current time is obtained. The target detection method is used to detect the moving object in the current frame image, and the target detection frame corresponding to the current frame image is obtained. The target part of the target detection frame is selected to obtain the part detection frame corresponding to the current frame image.

[0044] The target scene refers to a scene in which a mobile object can be captured by the mobile robot. The mobile object refers to an object that can move in the target scene, such as a moving pedestrian. The part detection box is used to label a target part of the mobile object. The target part can refer to a joint part of the mobile object, such as an ankle, a knee, or a shoulder. In some embodiments, the determination of the target part is determined by a part that can be detected by a radar sensor of the mobile robot. For example, since the mobile robot in this embodiment needs to identify a ground obstacle, the radar sensor can capture a range below, and therefore the target part can refer to the ankle of the mobile object. Correspondingly, the part detection box is located in the lower part of the target detection box, for example, it can be one third of the lower part of the target detection box. Alternatively, the part detection box can be determined based on the lower part of the target detection box. The part detection box includes the lower part of the target detection box but is not limited to the lower part of the target detection box. The target detection box is obtained by detecting the mobile object in the image captured by the vision sensor. As Figure 3 shown, the target detection box is used to frame the whole mobile object. The target detection box corresponds to one mobile object.

[0045] The vision sensor captures the target scene according to a preset capture frequency to obtain images at multiple time points.

[0046] In step S204, the point cloud captured by the radar sensor is projected onto the current frame image to obtain a set of projection points corresponding to the part detection box.

[0047] The radar sensor is a device that detects the position, speed, and other characteristics of an object using laser. The point cloud refers to data of two-dimensional or three-dimensional points obtained by detecting objects in the target scene by the radar sensor (as Figure 3 shown).

[0048] In this embodiment, the point cloud is data of points corresponding to obstacles in the target scene and data of points corresponding to target parts of the mobile object in the target scene captured by the radar sensor. The points corresponding to the target parts can be points corresponding to the ankles of the mobile object, or points corresponding to the lower legs of the mobile object.

[0049] The set of projection points refers to a set of projection points obtained by clustering the projection points located in the part detection box after projection. That is, one set of projection points corresponds to one cluster. In some part detection boxes, the number of sets of projection points is one. In some part detection boxes, the number of sets of projection points is two, or even more.

[0050] In this step, before projecting each point corresponding to the point cloud onto the current frame image, a transformation matrix between the visual sensor coordinate system corresponding to the visual sensor and the radar sensor coordinate system corresponding to the radar sensor can be obtained. Based on this transformation matrix, the radar coordinates of each point corresponding to the point cloud in the radar sensor coordinate system are then converted to image coordinates in the visual sensor coordinate system. This allows each point corresponding to the point cloud to be projected onto the current frame image. That is, the image coordinates of each point corresponding to the point cloud in the visual sensor coordinate system corresponding to the current frame image are obtained. Based on the image coordinates, each point corresponding to the point cloud can be annotated on the current frame image. A part detection frame is also annotated on the current frame image, so the positional relationship between each point corresponding to the point cloud and the part detection frame on the current frame image can be identified. Based on this positional relationship, a set of projected points corresponding to the part detection frame can be obtained. Exemplarily, obtaining the set of projected points corresponding to the part detection frame based on this positional relationship can include determining the projected points within the part detection frame based on this positional relationship and clustering these projected points to obtain the set of projected points.

[0051] By projecting each point corresponding to the point cloud onto the current frame image, the projection points corresponding to the part detection box are filtered out, thereby obtaining a set of projection points corresponding to the part detection box. In this way, the projection points outside the part detection box do not need to be processed, thereby reducing the amount of calculation and improving the running speed.

[0052] Step S206: Identify the target moving object from the moving objects.

[0053] The mobile objects are all objects that can move in the target scene. The target mobile object refers to the object to be locked and followed. The step of identifying the target mobile object from the mobile objects is based on the image collected by the visual sensor.

[0054] For example, the step of identifying the target mobile object from the mobile objects can occur in either the initial matching phase or the automatic matching phase. The initial matching phase refers to the phase in which the mobile robot matches the target object to be followed when the mobile robot is first started. The automatic matching phase refers to the phase in which the mobile robot identifies the target object to be followed during the following process.

[0055] In the initial matching stage, the target moving object in the target scene can be identified by the following steps: obtaining the key point information corresponding to the moving object in the current frame image; determining the candidate moving object based on the key point information corresponding to the moving object; obtaining the midpoint of the bottom side of the part detection box corresponding to the candidate moving object to obtain the coordinate position of the candidate moving object in the visual sensor coordinate system; converting the coordinate position to the robot coordinate system to obtain the corresponding local coordinate; and determining the candidate moving object closest to the mobile robot as the target moving object based on the local coordinates corresponding to each candidate moving object. The robot coordinate system can be a coordinate system with the position of the mobile robot as the origin.

[0056] In the automatic matching stage, the target moving object in the target scene can be identified by the odometer information of the mobile robot. Specifically, it can include: obtaining the last detection box corresponding to the moving object in the last frame image obtained at the last time; obtaining the current detection box corresponding to the moving object in the current frame image obtained at the current time; obtaining the odometer information of the mobile robot; converting the current detection box to the conversion detection box at the last time based on the odometer information; obtaining the association relationship between the last detection box in the last frame image and the current detection box in the current frame image based on the degree of overlap between the conversion detection box and the last detection box; and determining the target moving object in the current frame image based on the last detection box corresponding to the target moving object in the last frame image and the association relationship. The odometer information is used to record the movement information of the mobile robot (such as the movement direction and movement path at each time).

[0057] In step S208, the target projection point set corresponding to the target moving object is determined from the projection point set based on the part detection box corresponding to the target moving object.

[0058] The target projection point set refers to the projection point set corresponding to the target part of the detected target moving object.

[0059] In some cases, if the target moving object partially overlaps with other moving objects, the part detection box corresponding to the target moving object may include the projection point set corresponding to the target part of other moving objects, i.e., there can be more than two projection point sets in the part detection box corresponding to the target moving object. However, for the target moving object, only the projection point set corresponding to the target part of the target moving object can determine the position information of the target moving object. That is, if the target part is the leg, the number of target projection point sets is at most two. In this case, the target projection point set corresponding to the target moving object needs to be determined from the multiple projection point sets. This can reduce the influence of the point data of other moving objects on the determination of the object information of the target moving object and improve the accuracy of the determined object information of the target moving object.

[0060] In the embodiment, to determine the target projection point set, the projection point set corresponding to the part detection frame is filtered in combination with the part detection frame corresponding to the target moving object determined based on the visual sensor, to obtain the target projection point set corresponding to the target moving object.

[0061] For example, based on the part detection frame corresponding to the target moving object, the target projection point set corresponding to the target moving object is determined from the projection point set, which includes: determining the position of the target moving object based on the part detection frame corresponding to the target moving object; and determining the target projection point set based on the distance between the position of the target moving object and the projection point set.

[0062] Optionally, the position of the target moving object can be the midpoint of the bottom side of the part detection frame or the corner point of the part detection frame.

[0063] In step S210, the object information of the target moving object at the current time is obtained based on the target projection point set.

[0064] The object information refers to the position information, movement information, etc. of the target moving object.

[0065] For example, after the target projection point set is determined, i.e., the position of the target part of the target moving object is determined, the position information of the target moving object at the current time can be obtained based on the target projection point set. After the position information is obtained, the movement information of the target moving object, such as the moving speed, moving direction, moving path, etc., can also be obtained based on the position information at each time.

[0066] Optionally, to improve the accuracy of the object information of the target moving object, a preset filter (such as a Kalman filter) can be used to obtain the object information of the target moving object at each time. For example, after the position information of the target moving object at the current time is obtained based on the detected target projection point set, the measured position information is input into the preset filter, and the object information of the target moving object at the current time is updated based on the output of the preset filter.

[0067] In step S212, the following information of the mobile robot is obtained based on the object information, so that the target moving object is followed based on the following information.

[0068] The following information can include the motion path, motion direction, etc. of the mobile robot.

[0069] When deriving the mobile robot's following information based on object information, both the object information and the following information should be determined within the same reference coordinate system. To improve the accuracy of the following information, the reference coordinate system can be a global coordinate system, defined as a coordinate system whose origin is the initial position of the mobile robot after pairing with the target object. This global coordinate system can be used to record the motion information of the mobile robot since startup, as well as the movement information of the target object. This allows the mobile robot's following information to be derived in a global context, improving the stability and accuracy of the mobile robot's tracking.

[0070] It will be appreciated that the above steps can be performed by a processor built into the mobile robot. The processor receives data collected by multiple sensors and processes this data to ultimately obtain following information for the mobile robot. After obtaining the following information, the processor can transmit the following information to a control device of the mobile robot to control the movement of the mobile robot, thereby enabling the mobile robot to follow the target moving object.

[0071] In the target following method described above, a current frame image at the current moment is obtained through a built-in visual sensor of the mobile robot, and a part detection frame of a moving object in the target scene in the current frame image is obtained. Furthermore, a point cloud corresponding to the target scene at the current moment is obtained through a built-in radar sensor of the mobile robot. Each point corresponding to the point cloud is projected onto the current frame image to obtain a set of projection points corresponding to the part detection frame. In this way, a set of projection points corresponding to the moving objects in the target scene can be screened out. Then, a target moving object is identified from the moving objects. A set of target projection points corresponding to the target moving object is determined from the set of projection points based on the part detection frame corresponding to the target moving object. In this way, object information of the target moving object at the current moment can be obtained based on the set of target projection points corresponding to the target moving object. Based on the object information, the following information of the mobile robot can be obtained. Through the above steps, information related to the movement of the following target can be obtained based on the sensors originally built into the mobile robot, thereby saving costs. In addition, object information of the following target, such as position information, can be obtained by integrating multiple sensors. This can improve the accuracy of identifying the following target, reduce the probability of losing or mistracking, and thus improve the stability of target tracking.

[0072] In an exemplary embodiment, the visual sensor and radar sensor of the mobile robot are both able to identify the target moving object in the target scene. That is, the target moving object can be identified in the current frame image captured by the visual sensor, and the radar sensor can also capture the point cloud corresponding to the target part of the target moving object. In this case, the processor can combine the data collected by the visual sensor and the radar sensor to jointly determine the object information of the target moving object. For example, the target projection point set corresponding to the target moving object can be determined based on the detected part detection frame. Figure 4 As shown, based on the part detection frame corresponding to the target moving object, determining the target projection point set corresponding to the target moving object from the projection point set includes the following steps S402 to S406:

[0073] Step S402 : determining a reference point corresponding to the target moving object based on a part detection frame corresponding to the target moving object.

[0074] Among them, the reference point is a point located on the part detection frame. The reference point is data used to determine the object information of the target moving object in the dimension corresponding to the visual sensor. For example, if the target part is the leg (such as the ankle), the reference point can be the midpoint of the bottom edge of the part detection frame. If the target part is the left hand, and the corresponding part detection frame is located in the middle part of the overall detection frame corresponding to the moving object, the corresponding reference point can be the intersection of the left side and the bottom edge of the part detection frame, that is, the left corner point of the part detection frame.

[0075] Step S404: determining a set of candidate projection points located within a part detection frame corresponding to the target moving object from the set of projection points.

[0076] Among them, the projection point set is obtained after clustering the projection points within the part detection frame. That is to say, a projection point set is a projection point cluster. A preset proportion of projection points in a candidate projection point set are located within the part detection frame. For example, the coordinates of seventy percent of the projection points in a candidate projection point set are located within the coordinate range corresponding to the part detection frame. The coordinate range corresponding to the part detection frame is the coordinate range corresponding to the image area enclosed by the part detection frame. It can be understood that within a part detection frame, the number of projection point sets can be one, two, or even more. When there are multiple moving objects in the target scene, there are also multiple projection point sets. In order to determine the target projection point set corresponding to the target moving object, the projection point set corresponding to the target moving object, that is, the candidate projection point set, can be first filtered out from multiple projection point sets to distinguish the projection point sets corresponding to other moving objects. Exemplarily, the projection point set corresponding to the target moving object is filtered out from multiple projection point sets using the part detection frame corresponding to the target moving object. Reference Figure 5, the projection point set corresponding to the mobile object numbered 10 and the projection point set corresponding to the mobile object numbered 12 are detected. If the target mobile object is identified as the mobile object numbered 12, then the candidate projection point set is the three projection point sets within the part detection frame corresponding to the mobile object.

[0077] Exemplarily, after the target mobile object is identified from the plurality of mobile objects, the part detection frame corresponding to the target mobile object is identified from the plurality of part detection frames. Then the projection point set within the part detection frame corresponding to the target mobile object is determined as the candidate projection point set.

[0078] In step S406, the target projection point set is determined from the candidate projection point set based on the distance between the reference point and the candidate projection point set.

[0079] There can be more than two candidate projection point sets within the part detection frame corresponding to the target mobile object, or due to the measurement error of the radar sensor, the candidate projection point set within the part detection frame corresponding to the target mobile object does not include the target projection point set, but the projection point set corresponding to the obstacle close to the target mobile object.

[0080] To solve or improve the above problems, the target projection point set can be determined from the candidate projection point set based on the distance between the reference point and the candidate projection point set. Exemplarily, the cluster center point corresponding to each candidate projection point set is calculated. The distance between the cluster center point and the reference point is calculated. If the distance between the cluster center point and the reference point is greater than a preset threshold, it means that the candidate projection point set may not belong to the target mobile object, and the corresponding candidate projection point set is not the target projection point set. The candidate projection point set corresponding to the cluster center point and the reference point between which the distance is less than or equal to the preset threshold is determined as the target projection point set.

[0081] It can be understood that the number of target projection point sets is at most two. If the number of candidate projection point sets screened based on the distance exceeds two, the two or one candidate projection point set corresponding to the smallest distance is determined as the target projection point set. Please refer again to Figure 5 After calculating the distance between the reference point and the candidate projection point set, the first candidate point set within the mobile object numbered 12 is screened out, and the second candidate point set and the third candidate point set within the mobile object numbered 12 are determined as the target projection point set.

[0082] In the embodiment, the reference point corresponding to the target mobile object is determined based on the part detection box corresponding to the target mobile object, the candidate projection point set located in the part detection box corresponding to the target mobile object is determined from the projection point set, and the target projection point set is determined from the candidate projection point set based on the distance between the reference point and the candidate projection point set. The projection point set corresponding to the target object can be obtained, so that the accuracy of the obtained object information can be improved.

[0083] In one of the embodiments, the target projection point set can be determined from the candidate projection point set based on the distance between the reference point and the candidate projection point set by the following steps: calculating the center point corresponding to each candidate projection point set; calculating the distance between each center point and the reference point; determining the target distance meeting the first target condition from the distances; and determining the candidate projection point set corresponding to the target distance in the candidate projection point set as the target projection point set.

[0084] In the embodiment, the first target condition is used to limit the selected target distance to be less than a preset threshold, and the selected target distance is less than the remaining distances. The number of target distances is at least one. Alternatively, the number of target distances can be determined based on the type of the target part. If the target part is an ankle, the number of target distances is two. If the target part is a brain, the number of target distances is one.

[0085] In the embodiment, the center point corresponding to one candidate projection point set can represent the average or central position of all points in the candidate projection point set. When calculating the distance between each center point and the reference point, the Euclidean distance value between each center point and the reference point can be calculated.

[0086] In the embodiment, the distance between the reference point and the center point is calculated, and the target projection point set is selected from the candidate projection point set based on the calculated distance and the preset threshold. The distance between each candidate projection point set and the reference point can be accurately obtained, so that the target projection point set corresponding to the target mobile object can be accurately obtained, the interference caused by the projection point set of other objects can be removed, and the accuracy of the subsequent obtained object information can be improved.

[0087] In some embodiments, in the process of projecting the point cloud collected by the radar sensor to the current frame image to obtain the projection point set corresponding to the part detection box, the mapping relationship between the visual sensor and the radar sensor can be obtained based on the parameters of the visual sensor and the parameters of the radar sensor; the projection point position corresponding to each point of the point cloud in the current frame image is obtained based on the mapping relationship, and the plurality of projection points corresponding to the point cloud is obtained; the projection points located in the part detection box are marked as object projection points; and the object projection points are clustered to obtain the projection point set corresponding to the part detection box.

[0088] The mapping relationship between the visual sensor and the radar sensor refers to a correlation relationship between a visual sensor coordinate system corresponding to the visual sensor and a radar sensor coordinate system corresponding to the radar sensor.

[0089] The radar sensor emits laser in a preset height range, and the laser is reflected back to the radar sensor after contacting an object, so that the radar sensor can detect the object in the preset height range. In addition to the moving object, there can be other objects in the target scene. Referring to Figure 5 , for example, stationary objects, the radar sensor can collect both projection point sets corresponding to moving objects and projection point sets corresponding to other objects in addition to moving objects.

[0090] Exemplarily, based on the mapping relationship, the points collected by the radar sensor at the current time can be mapped to the matching current frame image. Then, the points mapped to the inside of the part detection frame can be clustered to obtain the projection point set corresponding to each part detection frame in the current frame image.

[0091] Optionally, after obtaining the projection point set corresponding to each part detection frame, the projection point set can also be filtered to screen out some projection point sets. The specific filtering process can include: obtaining the clustering attribute corresponding to the projection point set; removing the projection point set corresponding to the clustering attribute that does not satisfy the preset condition to update the projection point set at the current time.

[0092] The clustering attribute can include: a proportion of the number of points in the projection point set inside the part detection frame to the number of points in the projection point set, and a length of the minimum circumscribed matrix corresponding to the projection point set. The corresponding preset condition is that the length of the minimum circumscribed matrix corresponding to the projection point set is less than a preset length and the above proportion (the proportion of the number of points in the projection point set inside the part detection frame to the number of points in the projection point set) is greater than or equal to a preset proportion.

[0093] In this embodiment, by projecting the point cloud collected by the radar sensor to the current frame image, the point cloud can be screened based on the part detection frame corresponding to the current frame image, so that the number of clustered points can be reduced, the data amount of clustering can be reduced, and the speed of following can be improved, and the possibility of robot downtime can be reduced.

[0094] In actual applications, the visual sensor of the mobile robot can not always detect the part detection frame corresponding to the target moving object. For example, the target moving object is blocked by other moving objects when walking, so that the visual sensor cannot detect the overall detection frame corresponding to the target moving object, and thus the part detection frame corresponding to the target moving object cannot be obtained. In this case, if the radar sensor can still detect the set of projection points, the object information of the target moving object can be obtained based on the last frame of image obtained at the last time. In some embodiments, if the part detection frame of the moving object in the target scene in the current frame of image at the current time is not obtained, the set of projection points is obtained based on the point cloud, the center point corresponding to the set of projection points is calculated, the last frame position of the target moving object at the last time is obtained, and the set of target projection points is determined from the set of projection points based on the distance between the last frame position and the center point.

[0095] The last time is earlier than the current time. The last time is the time immediately preceding the current time. It can be understood that at the current time, the object information of the target moving object at the last time is known. To ensure the consistency of the obtained object information, the coordinate system based on which the object information is obtained can be a global coordinate system. The global coordinate system can be a coordinate system constructed with a position at which the mobile robot establishes an association with the target moving object as a coordinate origin.

[0096] The last frame position obtained in the present embodiment is determined based on the global coordinate system. Therefore, in the process of calculating the distance between the last frame position and the center point, the coordinate information of the center point can be converted to the global coordinate system. This involves the association relationship between the coordinate system corresponding to the radar sensor and the global coordinate system. The Euclidean distance between the center point and the last frame position is then calculated in the global coordinate system. Among the calculated multiple Euclidean distances, the distances greater than a preset threshold are removed, and then the smallest preset number of Euclidean distances are selected from the distances greater than the preset threshold. The set of projection points corresponding to the smallest preset number of Euclidean distances is determined as the set of target projection points.

[0097] In the present embodiment, the last frame position of the target moving object at the last time is obtained, and the set of target projection points of the target moving object is determined from the set of projection points based on the distance between the last frame position and the center point corresponding to the set of projection points. In this way, the position of the target moving object can be obtained when the target moving object cannot be determined by recognizing the detection frame. This takes into account various situations in the actual following process and performs corresponding processing, thereby improving the stability and comprehensiveness of the target moving object.

[0098] In a possible implementation, to improve the correctness of the target projection point set, the position of the target moving object at the current time can be predicted based on previously obtained data (for example, the object information at the previous time). Then, the target projection point set is determined in combination with the predicted position and the position of the previous frame at the previous time. To be able to predict the position of the target moving object at the current time, a Kalman filter can be used. As shown in FIG. 6, determining the target projection point set from the projection point set based on the distance between the position of the previous frame and the center point includes steps S602 to S610. Wherein: Figure 6

[0099] In step S602, the position of the target moving object at the current time is predicted based on the position of the previous frame corresponding to the target moving object, to obtain a predicted position.

[0100] The position of the previous frame can be obtained based on the object information corresponding to the target moving object obtained at the previous time. For example, a preset filter can predict the object information at the next time based on the object information at the current time. Therefore, after obtaining the object information at the previous time, the preset filter can predict the object information at the current time, that is, can predict the position of the target moving object at the current time, that is, the predicted position.

[0101] It can be understood that the predicted position at the current time is also determined based on the global coordinate system.

[0102] In step S604, the first distance between the position of the previous frame and each center point is calculated.

[0103] That is, in the global coordinate system, the first distance between the position of the previous frame and the center point corresponding to each projection point set is calculated. The first distance can be the Euclidean distance between the position of the previous frame and the center point.

[0104] In step S606, a candidate projection point set corresponding to the target distance is selected from the projection point set.

[0105] The target distance is selected from at least one first distance. The target distance is smaller than other distances in the first distance. The number of target distances is at least one. For example, the target distance is the smallest two first distances in the first distance.

[0106] For example, since multiple projection point sets can be collected at the current time, a candidate projection point set corresponding to the target distance can be selected from the multiple projection point sets. That is, the closest preset number of projection point sets to the position of the previous frame are selected from the multiple projection point sets. For example, the closest two projection point sets to the position of the previous frame are selected from the multiple projection point sets to obtain the candidate projection point set.

[0107] ​Step S608, a second distance between the center point corresponding to the candidate projection point set and the predicted position is calculated.

[0108] Exemplarily, after obtaining the candidate projection point set, a second distance between each candidate projection point set and the predicted position is calculated. The second distance is obtained based on the Euclidean distance between the center point corresponding to the candidate projection point set and the predicted position.

[0109] Step S610, if the second distance is less than a preset distance, the candidate projection point set corresponding to the second distance is determined as the target projection point set.

[0110] Exemplarily, if the candidate projection point set is far away from the position of the predicted target moving object, it indicates that the candidate projection point set should not be the projection point set corresponding to the target moving object. Therefore, after obtaining the second distance, each obtained second distance is screened based on the preset distance. If the second distance is less than or equal to the preset distance, the candidate projection point set corresponding to the second distance is determined as the target projection point set. If the second distance is greater than the preset distance, the candidate projection point set corresponding to the second distance is not the target projection point set.

[0111] In this embodiment, by combining the last frame position of the target moving object at the last time and the predicted position corresponding to the target moving object, the target projection point set corresponding to the target moving object can be determined, which can improve the accuracy of the obtained target projection point set.

[0112] After the above steps, the target projection point set corresponding to the target moving object at the current time can be determined, so that the object information of the target moving object at the current time can be obtained based on the target projection point set. In some cases, the radar sensor can detect two legs of the target moving object, so the number of target projection point sets can be two. In other cases, the radar sensor can only detect one leg of the target moving object, so the number of target projection point sets is only one. The method of obtaining object information is different in different cases.

[0113] When the number of target projection point sets is two, the first center points corresponding to the two target projection point sets are calculated, the first coordinates of a midpoint between the two first center points in the current frame image are obtained, the first coordinates corresponding to the midpoint are converted to the odometry coordinate system, the first global position corresponding to the midpoint is obtained, and the current position of the target moving object in the target scene is determined based on the first global position.

[0114] When the number of the target projection point set is one, a second center point corresponding to the target projection point set is calculated, a second coordinate of the second center point under the current frame image is obtained, the second coordinate is converted to the odometer coordinate system, a second global position of the second coordinate under the odometer coordinate system is obtained, and a current position of the target moving object in the target scene is determined based on the second global position.

[0115] The odometer coordinate system is the global coordinate system described above.

[0116] Optionally, after the target projection point set is obtained, the target projection point set can be first mapped to the odometer coordinate system. Then, based on coordinate information of each point included in the target projection point set under the odometer coordinate system, a center point corresponding to the target projection point set is determined. Then, object information of the target moving object is determined based on the center point. When there are two center points, coordinate information of a midpoint between the two center points is determined as the current position of the target moving object at the current time. When there is only one center point, coordinate information of the center point is determined as the current position of the target moving object at the current time.

[0117] In some embodiments, the data of the points collected by the radar sensor has high accuracy, and the object information determined based on the target projection point set also has high accuracy. Therefore, the object information of the target moving object can be directly determined based on the current position obtained from the target projection point set. In other embodiments, the accuracy of the data of the points collected by the radar sensor can not be very high, and a filter can be used to adjust the current position to improve the accuracy of the finally determined object information. Illustratively, determining the object information of the target moving object at the current time based on the current position includes inputting the current position to a preset filter, and updating the current position of the target moving object at the current time based on output information of the preset filter.

[0118] It can be understood that the current speed of the target moving object at the current time can also be obtained based on the output information.

[0119] In this embodiment, the final object information of the target moving object is obtained by using the preset filter, which can reduce the probability of inaccurate object information caused by possible measurement errors of the radar sensor, thereby improving the accuracy of the object information determined at each time.

[0120] In actual application, at the current time, the radar sensor can not obtain the projection point set corresponding to the moving object in the target scene. For example, the target part of the moving object in the target scene is all blocked, and the projection point set corresponding thereto cannot be obtained. In this case, the radar sensor can be controlled to continue collecting the point cloud corresponding to the target scene within a preset time. If the target projection point set cannot be obtained based on the collected point cloud within the preset time, the mobile robot is controlled to issue a preset prompt information to prompt the target moving object. If the target projection point set is obtained again based on the collected point cloud within the preset time, the object information of the target moving object is obtained based on the latest target projection point set, and the mobile robot is controlled to follow the target moving object based on the object information.

[0121] In some embodiments, at the current time, the visual sensor can recognize the target moving object in the target scene, but at this time, the target part of the moving object can be blocked by some obstacles, so that the radar sensor can not detect the target projection point set of the target moving object based on the collected point cloud. Therefore, the object information of the target moving object cannot be determined based on the point cloud. In this case, the mobile robot can re-detect the target projection point set of the target moving object, re-obtain the position of the target moving object, and re-follow the target.

[0122] In some embodiments, at the current time, the radar sensor can detect the target projection point set, but at this time, the target moving object can move to the shooting range of the visual sensor, so that the visual sensor cannot recognize the target moving object (this is because the detection range of the radar sensor is larger than the shooting range of the visual sensor). In this case, the mobile robot can re-shoot the target moving object, determine the target projection point set corresponding to the target moving object from the projection point set based on the part detection box corresponding to the target moving object, obtain the object information of the target moving object at the current time based on the target projection point set, and obtain the following information of the mobile robot based on the object information, so as to follow the target moving object based on the following information.

[0123] In some embodiments, the data collected by the visual sensor can be verified based on the data collected by the radar sensor. For example, if the visual sensor detects a part detection box corresponding to the target moving object based on the collected data, but the distance between the target candidate point set determined by the radar sensor based on the collected data and the reference point corresponding to the part detection box exceeds a preset value, it indicates that the detected target moving object is incorrect at the detection level of the visual sensor. When the verification result is not matched, a prompt information of target loss can be generated. After the target loss, the target moving object can make a preset action to re-establish the association with the mobile robot.

[0124] In some embodiments, the data collected by the radar sensor can be verified based on the data collected by the visual sensor. For example, after the radar sensor collects some point clouds, the analysis obtains a target projection point set corresponding to the target moving object, but the visual sensor does not detect a moving object in the current frame image collected at the current time. It is indicated that the target projection point cloud obtained by analysis is incorrect. For example, the leg of a chair in the target scene can be detected as the leg of a moving object. When the verification result is not matched, a target loss prompt information can be generated. When the verification result is matched, the object information of the target moving object at the current time is obtained based on the obtained target projection point set, and the following information of the mobile robot is obtained based on the object information, so as to follow the target moving object based on the following information.

[0125] Through the above embodiments, various problems that can occur in the actual application of the mobile robot following the target moving object are comprehensively considered, and solutions to these problems are designed, so that the stability and comprehensiveness of the mobile robot following the target moving object can be improved, thereby improving the following experience. And through the mutual verification of the data between multiple sensors, the accuracy of the final determined following target position can be improved, thereby improving the accuracy of the following target.

[0126] It can be understood that before the mobile robot automatically follows the target moving object, the target moving object needs to be associated with the mobile robot, so that the mobile robot can automatically identify the target moving object in the subsequent process. In some embodiments, the target moving object can make some special actions, so that the mobile robot can filter the target moving object from multiple moving objects. The special action can be, for example, raising a hand. That is, the target moving object can establish an association with the mobile robot by raising a hand. The special action of the target moving object can be determined by identifying the positional relationship between the key points of the target moving object. Therefore, in the process of initially establishing the association, the target moving object is identified from the moving objects, including: obtaining key point information corresponding to the moving object in the current frame image; determining a candidate moving object based on the key point information corresponding to the moving object; obtaining a midpoint of the bottom side of the part detection box corresponding to the candidate moving object to obtain a coordinate position of the candidate moving object in the visual sensor coordinate system; converting the coordinate position to the robot coordinate system to obtain a corresponding local coordinate; and determining the candidate moving object closest to the mobile robot as the target moving object based on the local coordinates corresponding to each candidate moving object.

[0127] The robot coordinate system is a coordinate system constructed with the position of the robot as the origin. The key point information can be the information of the key points of the moving object obtained based on a target detection algorithm (for example, YOLOv8) (for reference Figure 7). The to-be-selected moving object refers to a moving object whose corresponding key point information meets a preset action. The preset action is the special action described above.

[0128] Optionally, determining the to-be-selected moving object based on the key point information of the moving object comprises: obtaining, based on the key point information, first key point coordinates of a first target part and second key point coordinates of a second target part of the moving object; and determining the moving object as the to-be-selected moving object if the longitudinal coordinate of the first key point coordinates is greater than the longitudinal coordinate of the second key point coordinates. If the preset action is raising a hand, the first target part can be a wrist and the second target part can be an ear.

[0129] Exemplarily, after the to-be-selected moving object is determined, a midpoint of the bottom edge of the part detection frame corresponding to the to-be-selected moving object is obtained, so that the coordinate position of the to-be-selected moving object in the visual sensor coordinate system can be obtained. Since the number of the recognized to-be-selected moving objects can be more than one, and the target moving object is usually closer to the mobile robot when the target moving object is associated with the mobile robot, the distances between each to-be-selected moving object and the mobile robot can be compared to determine the target moving object.

[0130] Specifically, the coordinate position of the to-be-selected moving object in the visual sensor coordinate system can be converted to the robot coordinate system to obtain the corresponding local coordinate. Based on the local coordinate of the to-be-selected moving object, the distance between the to-be-selected moving object and the mobile robot is calculated. The to-be-selected moving object closest to the mobile robot is determined as the target moving object.

[0131] It can be understood that after the target moving object is determined, the target projection point set corresponding to the target moving object can be determined from the projection point set based on the part detection frame corresponding to the target moving object; the target projection point set is mapped to the odometer coordinate system, the center point of the target projection point set in the odometer coordinate system is calculated, so that the initial position of the target moving object at the start of the start is obtained. The preset filter can be established based on the initial position, so as to facilitate the subsequent prediction of the object information of the target moving object by the preset filter.

[0132] In this embodiment, the posture of the moving object in the target scene is obtained by obtaining the key point information of the moving object, so that the target moving object can be recognized from a plurality of moving objects and associated with the target moving object, and the purpose of following the target moving object is achieved.

[0133] In the subsequent process of automatically identifying the target moving object by the mobile robot, the situation of following the wrong target due to the rotation of the robot itself can occur. Referring to Figure 7From time t-1 to time t, the robot rotates to the left, resulting in the pedestrian detection frame originally numbered 1 having a very small IOU (Intersection over Union) at time t-1, and a very large IOU with the new one, resulting in a wrong match. Similarly, the pedestrian detection frame numbered 2 at time t-1 matches the pedestrian detection frame numbered 1 at time t, and the pedestrian detection frame numbered 3 at time t-1 matches the pedestrian detection frame numbered 2 at time t.

[0134] To solve or improve the above-mentioned problem, the target moving object in the current frame image can be determined based on the mobile robot's odometer, thereby identifying the target moving object in each frame image. In an exemplary embodiment, mileage information collected by the mobile robot's odometer can be obtained; based on the mileage information, the degree of overlap between each detection frame corresponding to the current frame image and each detection frame corresponding to the previous frame image can be determined; from the various detection frames corresponding to the current frame image, the detection frame with the highest degree of overlap with the detection frame corresponding to the target moving object in the previous frame image is determined; and this detection frame is determined to be the detection frame corresponding to the target moving object in the current frame image. This can solve the problem of multi-target tracking errors caused by the robot's rotation, thereby correctly identifying the target moving object.

[0135] Alternatively, as Figure 8 As shown, identifying a target moving object from moving objects includes steps S802 to S812.

[0136] Step S802 : obtaining a previous detection frame corresponding to the previous frame of image of the moving object obtained at the previous moment.

[0137] The number of moving objects that appeared in the target scene at the last moment is the same as the number of the last detection frame.

[0138] Step S804: Obtain a current detection frame corresponding to the moving object in the current frame image.

[0139] The number of moving objects appearing in the target scene at the current moment is the same as the number of current detection frames.

[0140] It is understood that the positions of the current and previous detection frames in steps S802 and S804 are determined based on the visual sensor coordinate system. Since the origin of the visual sensor coordinate system is based on the robot's position, the visual sensor coordinate system is different at different times. Accordingly, the current and previous detection frames are not in the same dimension. If you need to compare the current and previous detection frames, you can convert the current detection frame's position to the visual sensor coordinate system corresponding to the previous detection frame to ensure consistency.

[0141] At step S806, the odometer information of the mobile robot is obtained.

[0142] The odometer information can include information related to the movement state of the mobile robot since the start. For example, the odometer transformation matrix at each time point can be obtained. The odometer transformation matrix is used to describe the transformation relationship of the robot from one position to another position. This matrix is usually a homogeneous transformation matrix, which can represent the rotation information and translation information of the robot.

[0143] At step S808, the current detection frame is converted to the converted detection frame corresponding to the previous detection frame based on the odometer information.

[0144] The converted detection frame is a detection frame obtained by converting the current detection frame to the visual sensor coordinate system corresponding to the previous detection frame.

[0145] For example, based on the odometer information of the mobile robot, the transformation relationship between the odometer information at different time points can be obtained. Based on the transformation relationship, the information corresponding to the second time odometer information in the first time first odometer coordinate system can be obtained. The second time is later than the first time. The first odometer coordinate system takes the position of the robot at the first time as the origin. For example, based on the data collected by the odometer, the transformation matrix of the odometer at t-1 time and the transformation matrix of the odometer at t time are obtained. Then, based on the transformation matrix at t-1 time and the transformation matrix at t time, the above transformation relationship is obtained, that is, the value of the odometer at t time in the first odometer coordinate system at t-1 time.

[0146] After obtaining the above transformation relationship, the image coordinate information corresponding to the image coordinate information at the first time in the first visual sensor coordinate system at the second time can be obtained. The first visual sensor coordinate system takes the position of the robot at the first time as the origin. Then, the current detection frame detected at the current time can be mapped to the visual sensor coordinate system corresponding to the previous detection frame detected at the previous time, to obtain the converted detection frame corresponding to the current detection frame. In this way, the converted detection frame and the previous detection frame are in the same visual sensor coordinate system, and thus the converted detection frame and the previous detection frame can be compared.

[0147] In this step, based on the odometer information of the mobile robot, the matching degree of the current detection frame and the converted detection frame can be improved, thereby improving the accuracy of the overlap degree in subsequent calculation.

[0148] At step S810, based on the overlap degree between the converted detection frame and the previous detection frame, the association relationship between the previous detection frame in the previous frame image and the current detection frame in the current frame image is obtained.

[0149] After the above steps, the current detection frame has been mapped to the visual sensor coordinate system at the previous time. That is, the current detection frame has been traced back to obtain the corresponding detection frame at the previous time. Therefore, the mapping obtained conversion detection frame is compared with the previous detection frame, that is, the association relationship between the conversion detection frame and the previous detection frame can be determined, and the association relationship between the current detection frame and the previous detection frame can be obtained.

[0150] Exemplarily, in the process of comparing the mapping obtained conversion detection frame with the previous detection frame, the degree of coincidence between each conversion detection frame and each previous detection frame can be obtained. Then, based on the degree of coincidence, the association relationship between the plurality of conversion detection frames and the plurality of previous detection frames can be determined. It can be understood that one conversion detection frame corresponds to one previous detection frame. There are some previous detection frames without corresponding conversion detection frames.

[0151] Step S812, based on the previous detection frame corresponding to the target moving object in the previous frame image and the association relationship, determining the target moving object in the current frame image.

[0152] It can be understood that at the previous time, the target moving object is determined. For example, at the initial time, the target moving object is determined by recognizing the posture of the moving object. At each time thereafter, the information of the target moving object at the current time can be determined based on the information of the target moving object at the previous time.

[0153] Exemplarily, after obtaining the above association relationship, the previous detection frame corresponding to the target moving object is determined, the conversion detection frame corresponding to the previous detection frame is determined based on the association relationship, and the corresponding current detection frame is determined based on the conversion detection frame, so that the target moving object in the target scene can be determined based on the current detection frame.

[0154] In this embodiment, the change of the odometer information of the mobile robot at different times reflects the change of the position of the moving object at different times, so that when the position of the moving object in the visual sensor coordinate system at the current time is obtained, the position of the moving object in the visual sensor coordinate system at the previous time can be determined, and then the target moving object in the current frame image can be obtained based on the position and the target moving object in the previous frame image. In this way, the moving object at the current time can be traced back based on the information of the odometer of the robot, so that the problem of matching error can be solved.

[0155] In some embodiments, the step of obtaining the odometry information of the mobile robot can include: obtaining last odometry information of the mobile robot at a last time and current odometry information of the mobile robot at a current time. The converting the current detection box to the converted detection box at the last time based on the odometry information includes: obtaining a current reference point of the moving object at the current time; obtaining a target transformation matrix based on the last odometry information and the current odometry information; mapping the current reference point to a transformed reference point corresponding to the current reference point based on the target transformation matrix and based on a visual sensor coordinate system at the last time; obtaining size information of the detection box of the moving object at the current time; and obtaining the converted detection box of the moving object at the last time based on the size information and the transformed reference point.

[0156] The target transformation matrix is used to represent a transformation relationship of image coordinate information at the current time relative to image coordinate information at the last time. The size information can include a length and a width of the detection box. The current reference point is a point used to represent a position of the detection box. The description of the current reference point can refer to the description of the reference point above.

[0157] For example, the current reference point of the moving object at the current time is obtained first. It can be understood that the current reference point is determined based on a current visual sensor coordinate system. The current visual sensor coordinate system takes a position of the mobile robot at the current time as an origin. For example, a bottom edge midpoint of each pedestrian detection box in an RGB image at time t is obtained as a landing point of the pedestrian. In this way, a position of the pedestrian in a robot coordinate system at time t can be obtained.

[0158] Then, based on the previous odometry information and the current odometry information, a target transformation matrix is obtained. Specifically, based on the previous odometry information and the current odometry information, a value of the current odometry at the current time is obtained in a target value corresponding to the previous odometry coordinate system. Then, based on the current reference point and the aforementioned target value, the target transformation matrix is obtained. For example, the previous odometry information can be a transformation matrix of the t-1 time odometry, and the current odometry information can be a transformation matrix of the t time odometry. The transformation matrix of the t-1 time odometry is Twr-1, and the transformation matrix of the t time odometry is Twr. In the t-1 time odometry coordinate system, the value of the t time odometry is denoted as Tr-1r. According to the formula Tr-1r=Tr-1w*Twr, the value of Tr-1r can be obtained, where Tr-1w is the inverse matrix of Twr-1. The bottom edge midpoint of each pedestrian detection box in the t time RGB image is obtained, which is the landing point of the pedestrian. That is, the bottom edge midpoint of the detection box can represent the projection point of the pedestrian on the ground. It can be known that the position of the person at this time in the robot coordinate system is Trp. Then, according to Tr-1p=Tr-1r*Trp, the transformation matrix of each pedestrian at the t-1 time vision sensor coordinate system at the t time, that is, the target transformation matrix, can be obtained. The vision sensor coordinate system is constructed with the current position of the mobile robot as the origin.

[0159] Based on the target transformation matrix, the current reference point is mapped to the vision sensor coordinate system based on the previous time to obtain the transformation position corresponding to the current reference point. The target transformation matrix represents the transformation reference point between the vision sensor coordinate system at the current time and the vision sensor coordinate system at the previous time. Therefore, the current reference point can be mapped to the vision sensor coordinate system at the previous time to obtain the transformation reference point corresponding to the current reference point. For example, the coordinate position of the current reference point is (200, 300), and the coordinate position of the mapping to the vision sensor coordinate system at the previous time is (190, 290).

[0160] Then, the size information of the detection box of the moving object at the current time is obtained. Based on the size information and the transformation reference point, the corresponding conversion detection box of the moving object in the previous frame image is obtained. The size information includes the length and width of the detection box. Alternatively, based on the size information and the transformation reference point, the step of obtaining the conversion detection box of the moving object in the previous frame image can include: extending a target length to both ends of the transformation reference point to obtain the bottom edge of the detection box; extending a target width in a target direction from the starting point of the end point of the bottom edge to obtain the side edge of the detection box. The target length is half of the length of the detection box, and the target width is the width of the detection box. The target direction is perpendicular to the bottom edge and upward. Alternatively, if the transformation reference point corresponds to the end point of the edge of the detection box, the bottom edge is obtained by extending a target length to the transformation reference point, where the target length is the length of the detection box.

[0161] In this embodiment, the current reference point of the moving object at the current time is obtained, and a target transformation matrix is obtained based on the last odometry information and the current odometry information. In this way, the conversion relationship between the target moving object at each time is obtained based on the motion information of the robot, that is, the detection boxes of the target moving object at each time can be associated, so that the identification of the wrong target moving object caused by the motion such as rotation of the robot can be reduced. Then, based on the target transformation matrix, the current reference point is mapped to the coordinate system based on the last time, to obtain a transformation reference point corresponding to the current reference point, to obtain the size information of the detection box of the moving object at the current time, and based on the size information and the transformation reference point, to obtain the corresponding conversion detection box of the moving object in the last frame image. In this way, the information of the target moving object at the last time is traced back through the information of the target moving object at the current time, so as to improve the accuracy of identifying the target moving object.

[0162] After obtaining the object information of the target moving object at the current time, the following information of the moving robot can be obtained based on the object information. For example, the following information of the moving robot based on the object information includes: obtaining the object position of the target moving object based on the object information; obtaining the following information of the moving robot based on the object position and the position of the moving robot. Wherein, the following information can include the moving direction, the moving speed and the moving path of the moving robot. As shown in the following figure, the following information of the moving robot based on the object information includes steps S902 to S910. Wherein: Figure 9

[0163] Step S902, obtaining the target object position of the target moving object based on the object information.

[0164] Step S904, obtaining the obstacle information at the current time, and obtaining the mileage position of the moving robot at the current time.

[0165] Wherein, the mileage position is the position of the moving robot in the odometry coordinate system. The odometry coordinate system is constructed based on the position of the moving robot as the coordinate origin after the moving robot recognizes the target moving object. That is, the odometry coordinate system is a global coordinate system. It can be understood that the action trajectory of the moving robot determined in the global coordinate system is reasonable and accurate.

[0166] Step S906, calculating the distance between the target object position and the mileage position.

[0167] Step S908, judging whether the distance is greater than the first preset distance.

[0168] Step S910, if the distance is less than or equal to the first preset distance, generating a brake instruction.

[0169] ​The brake instruction is used to control the mobile robot to stop moving. The generated brake instruction can be sent to the control device, so that the control device controls the mobile robot to stop moving.

[0170] In step S912, if the distance is greater than the first preset distance, the following information of the mobile robot is determined based on the obstacle information, the target object position, and the mileage position.

[0171] The first preset distance is a preset safety distance. The obstacle information refers to information related to obstacles in the target scene, such as position, contour, etc. The following information refers to information related to the movement of the mobile robot in the target scene for following the target moving object.

[0172] In this embodiment, the following path of the mobile robot is determined by obtaining the information of the obstacle, which can effectively avoid obstacles. By determining the following path of the mobile robot in the global odometer coordinate system, the accuracy of the following path can be improved, and it is beneficial to predict the following path by comprehensively considering the object information of the target moving object.

[0173] In some embodiments, during the process of the mobile robot following the target moving object, a short-term loss of following may occur. To deal with this situation, the following information of the mobile robot may change. For example, if the target moving object cannot be identified from the moving object, the disappearance position corresponding to the target moving object is obtained. If the distance between the disappearance position and the mileage position is greater than or equal to the second preset distance, the movement instruction is generated based on the disappearance position. If the distance between the disappearance position and the mileage position is less than the second preset distance, the moving direction of the target moving object is obtained, and the following information of the mobile robot is determined based on the moving direction.

[0174] The movement instruction is used to control the mobile robot to reach the disappearance position, so that the mobile robot can identify the target moving object at the disappearance position. The disappearance position refers to the position of the target moving object last detected by the mobile robot. The second preset distance is less than the first preset distance. The second preset distance is used for the process of target disappearance. That is, when the target moving object disappears far away from the mobile robot, the safety distance between the mobile robot and the target moving object can be shortened to detect the following target. For example, at the 10th second, the mobile robot detects the target moving object and obtains the position of the target moving object. At the 11th second, the target moving object may be blocked by other obstacles, resulting in failure to detect the target moving object. At this time, if the mobile robot calculates that the distance between the target moving object is more than 1m (corresponding to the second preset distance), the distance requirement of 1.5m (corresponding to the first preset distance) is no longer considered, and the mobile robot can move to the position of the target moving object detected at the 10th second. The target moving object in the target scene is continuously detected at this position.

[0175] Optionally, the direction of the target moving object when it is last detected can also be obtained to improve the probability of detecting the target moving object. After obtaining the direction and the position of the target moving object when it is last detected, the mobile robot is controlled to move to the position and face the direction to detect the target moving object.

[0176] In this embodiment, by controlling the robot to perform relevant movements when the target is lost, the situation of following the target disappearing can be coped with, and the stability of target following can be improved.

[0177] The above process is applied when the moving direction of the target moving object and the moving direction of the mobile robot are not opposite, and when the target is lost, the mobile robot needs to move to the position where the target moving object disappears to detect the target moving object again. However, in some embodiments, the reason why the mobile robot cannot identify the target moving object can be that the target moving object is too close to the mobile robot, for example, the target moving object stands in front of the mobile robot to put something on the mobile robot, so that the mobile robot cannot identify the whole of the target moving object, and thus cannot detect the target moving object. Therefore, in the aspect of obtaining the moving direction of the target moving object and determining the following information of the mobile robot based on the moving direction, if the moving direction is opposite to the moving direction of the mobile robot, a waiting instruction is generated to control the mobile robot to wait in place for a preset time, and after the preset time, the mobile robot is controlled to rotate in place to be able to identify the target moving object. If the moving direction is not opposite to the moving direction of the mobile robot, a moving instruction is generated based on the position of disappearance, and the moving instruction is used to control the mobile robot to reach the position of disappearance.

[0178] In this embodiment, by considering the moving direction of the following target, the mobile robot is controlled to wait in place when the following target is close to the mobile robot, so that the situation of the mobile robot colliding with the following target can be avoided.

[0179] In an exemplary embodiment, to achieve following the target, this embodiment is divided into three parts: the first part is to determine the following target, the second part is to determine the information of the following target, and the third part is to determine the following information of the mobile robot. The following is a detailed introduction to the three parts:

[0180] I. Determining the following target

[0181] When the target to be followed has been determined, to determine the follow target, a previous detection box corresponding to the mobile object in a previous frame image obtained at a previous time can be obtained first; a current detection box corresponding to the mobile object in a current frame image can be obtained; previous odometry information of the mobile robot at the previous time and current odometry information of the mobile robot at the current time can be obtained; a current reference point of the mobile object at the current time can be acquired; a target transformation matrix is obtained based on the previous odometry information and the current odometry information, the target transformation matrix being used to represent a transformation relationship of image coordinate information at the current time relative to image coordinate information at the previous time; the current reference point is mapped to a coordinate system based on the previous time based on the target transformation matrix, to obtain a transformed reference point corresponding to the current reference point; size information of the detection box of the mobile object at the current time is obtained; the size information and the transformed reference point are used to obtain a converted detection box corresponding to the mobile object in the previous frame image; a correlation relationship between the previous detection box in the previous frame image and the current detection box in the current frame image is obtained based on a coincidence degree between the converted detection box and the previous detection box; and the target mobile object in the current frame image is determined based on the previous detection box corresponding to the target mobile object in the previous frame image and the correlation relationship.

[0182] For example, the position of the current landing point (the above-mentioned current reference point) in the robot coordinate system (for example, the coordinates in the robot coordinate system at the current time) can be obtained through the parameters (internal parameters and external parameters) of the visual sensor. The RGB image at time t-1 and the odometry data at time t-1 are obtained, and the RGB image at time t and the odometry data at time t are obtained. The change of the odometry between time t-1 and time t is calculated. That is, the value of the odometry at time t in the odometry coordinate system at time t-1. The value of the odometry at time t in the odometry coordinate system at time t-1 is obtained based on the product of the transformation matrix of the odometry at time t-1 and the transformation matrix of the odometry at time t. Then the bottom midpoint of each target detection box in the RGB image at time t is obtained, and the position of the target in the robot coordinate system is known. The transformation matrix of each target at time t in the robot coordinate system at time t-1 is obtained based on the product of the position of each target in the robot coordinate system and the value of the odometry at time t in the odometry coordinate system at time t-1. The coordinates of the landing point of each target at time t-1 are obtained based on the transformation matrix of each target and the landing point of the target at time t. The length and width of the target detection box at time t are combined to convert each target detection box at time t back to time t-1. Then the target detection box at time t converted back and the target detection box at time t-1 are subjected to IOU calculation to solve the problem of matching error.

[0183] It can be understood that after determining the target matching of two frames of images, a follow number can be set for each target to determine the follow target faster. Moreover, the key point information of each target can also be obtained through the above steps.

[0184] Reference Figure 10 If the mobile robot collects the image of the target scene through the fisheye camera sensor, image preprocessing is also needed to obtain the effective range of the image and reduce the resources of subsequent image processing. The image preprocessing includes rotation, size adjustment, and cropping, etc. Then the image is input into the target detection model to obtain the target detection box and the key points of the target needed for subsequent use. Then the odometer is used to eliminate the matching error of the moving object caused by the rotation of the mobile robot according to the above steps. That is, the embodiment determines the following target from a plurality of moving objects by obtaining the image collected by the vision sensor, the parameters of the vision sensor, and the data collected by the odometer.

[0185] When the target to be followed is not determined, it is necessary to first establish an association with the target to be followed. To determine the target to be followed, the key point information of the moving object in the current frame image can be obtained; the candidate moving object is determined based on the key point information corresponding to the moving object; the midpoint of the bottom side of the part detection box corresponding to the candidate moving object is obtained to obtain the coordinate position of the candidate moving object in the vision sensor coordinate system; the coordinate position is converted to the robot coordinate system to obtain the corresponding local coordinate; based on the local coordinates corresponding to each candidate moving object, the candidate moving object closest to the mobile robot is determined as the target moving object.

[0186] That is, when no pedestrian is locked for following, the pedestrian with the hand raised closest to the robot is selected for locking. The position of the pedestrian with the hand raised closest to the robot in the robot coordinate system can be obtained by the grounding point described above. The two or one radar leg candidate clusters closest to the footstep box of this pedestrian are selected, and the radar leg candidate clusters are within a certain threshold distance from the grounding point of this pedestrian, and cannot be too far away. The midpoint of the positions of the two or one radar leg candidate clusters in the odometer coordinate system is determined as the initial coordinate of the locked following pedestrian in the odometer coordinate system, and the Kalman filter is established based on this. The Kalman filter maintains the position and speed of the locked following pedestrian in the odometer coordinate system; at the same time, the number of the pedestrian detection box on the RGB image is recorded as the number of the locked following pedestrian follow_id.

[0187] II. Information for determining the following target

[0188] (I) Both the vision sensor and the radar sensor can detect the following object

[0189] 1. Obtain the part detection box of the moving object in the target scene in the current frame image at the current time. The part detection box is used to mark the target part of the moving object. For example, referring to Figure 11According to each pedestrian detection frame, the following number and 17 key points of each pedestrian output by the first part, a leg frame of each pedestrian (the lower 1 / 3 part of the detection frame is a foot frame) can be obtained.

[0190] 2. Project the point cloud collected by the radar sensor to the current frame image to obtain a projection point set corresponding to the part detection frame;

[0191] Obtain a mapping relationship between the visual sensor and the radar sensor based on parameters of the visual sensor and parameters of the radar sensor, the visual sensor being used for photographing a target scene;

[0192] Obtain a projection point position of each point corresponding to the point cloud in the current frame image based on the mapping relationship, to obtain a plurality of projection points corresponding to the point cloud.

[0193] Mark the projection points located in the part detection frame as object projection points.

[0194] For example, project the radar points to the RGB image according to the camera internal and external parameters and the radar external parameters, and mark whether the radar points are in the foot frame of the pedestrian. It can be understood that after obtaining the point cloud, the point cloud can be preprocessed to filter discrete isolated points and voxel filtering (filter out very close radar points). In this way, the amount of subsequent calculation can be reduced.

[0195] 3. Cluster the object projection points to obtain a projection point set corresponding to the part detection frame. For example, perform segmentation clustering on all radar points, and calculate the center point of each cluster. Calculate the attributes of each cluster: the proportion of the number of points in the pedestrian foot frame to the entire cluster points, the distance of the cluster from the position of the followed locked pedestrian in the last frame, and the length of the minimum bounding rectangle of the cluster points. Then filter out the projection point set according to the following rules: the length of the minimum bounding rectangle of the cluster points is small enough and the proportion of the number of points in the pedestrian foot frame to the entire cluster points is large enough, or the length of the minimum bounding rectangle of the cluster points is small enough and the distance of the cluster from the position of the followed locked pedestrian in the last frame is close.

[0196] 4. Identify a target moving object from the moving objects.

[0197] Based on the part bounding box corresponding to the target moving object, a reference point corresponding to the target moving object is determined. A candidate projection point set located in the part bounding box corresponding to the target moving object is determined from the projection point set. A center point corresponding to each candidate projection point set is calculated. The distance between each center point and the reference point is calculated. A target distance meeting a first target condition is determined from each distance. The candidate projection point set corresponding to the target distance in the candidate projection point set is determined as a target projection point set. If the number of target projection point sets is two, a first center point corresponding to the two target projection point sets is calculated, a first coordinate of a midpoint between the two first center points in the current frame image is obtained, the first coordinate corresponding to the midpoint is converted to the odometry coordinate system, a first global position corresponding to the midpoint in the odometry coordinate system is obtained, and the current position of the target moving object in the target scene is determined based on the first global position. If the number of target projection point sets is one, a second center point corresponding to the target projection point set is calculated, a second coordinate of the second center point in the current frame image is obtained, the second coordinate is converted to the odometry coordinate system, a second global position of the second coordinate in the odometry coordinate system is obtained, and the current position of the target moving object in the target scene is determined based on the second global position.

[0198] That is, if the numbered pedestrian bounding box on the previously recorded locked following pedestrian RGB image can be found on this frame image, the closest two or one radar human leg candidate clusters belonging to the pedestrian footstep box are selected, and the distance between the radar human leg candidate cluster and the landing point of the pedestrian is within a certain threshold, which cannot be too far. The midpoint of the positions of the closest two or one radar human leg candidate clusters in the odometry coordinate system is determined as the position of the locked following pedestrian in the odometry coordinate system. The position is used to update the Kalman filter subsequently to update the state (position and speed) of the following pedestrian.

[0199] 5. The current position is input to a preset filter, and the current position of the target moving object at the current time is updated based on the output information of the preset filter.

[0200] (ii) The visual sensor cannot detect the following object, but the radar sensor can detect the following object

[0201] If the bounding box of the part of the moving object in the target scene in the current frame image at the current time is not obtained, a set of projection points is obtained based on the point cloud, and a center point corresponding to the set of projection points is calculated. The position of the target moving object at the previous time is obtained, the position of the target moving object at the current time is predicted based on the position of the target moving object at the previous time, and a predicted position is obtained. The first distance between the position at the previous time and each center point is calculated. A candidate set of projection points corresponding to the target distance is selected from the set of projection points, and the target distance is the smallest preset number of first distances in the first distances. The second distance between the center point corresponding to the candidate set of projection points and the predicted position is calculated. If the second distance is less than or equal to a preset distance, the candidate set of projection points corresponding to the second distance is determined as the target set of projection points.

[0202] The steps after determining the target set of projection points are similar to the steps corresponding to the process in which both the visual sensor and the radar sensor can detect the following object, and will not be described here.

[0203] (III) The visual sensor can detect the following object but the radar sensor cannot detect the following object

[0204] If the radar sensor has not detected the following object for a preset time, it is considered that the target has been lost. If the radar sensor detects the following object within the preset time, the object information of the target following object is determined again based on the target set of projection points of the detected following object. That is, if at least one radar leg cluster cannot be found, it is considered that the target following pedestrian is lost at the radar level. If the target is lost for 5 consecutive frames, it is considered that the pedestrian has been lost. If at least one radar leg cluster can be found within the 5 frames of loss, it is considered that the target has been recovered.

[0205] In short, the steps of the process are implemented as follows:

[0206] 1. Lock the following person by raising the hand, and re-lock after losing the target by recognizing the raised hand.

[0207] 2. The radar point cloud is projected onto the image by the relative position relationship between the radar and the RGB.

[0208] 3. The point cloud clusters the object, which is the two legs of a pedestrian, and can be easily separated from other point clouds in the environment, reducing misidentification.

[0209] 4. The leg part of the pedestrian can be obtained by identifying the pedestrian frame in the image, so that the image pedestrian id can be associated with the radar cluster of the clustered leg of the radar.

[0210] 5. Multiple target following can be performed through the first part.

[0211] 6. There are also many small clusters on the laser radar that look like human legs, such as stool legs, etc. Through the steps of calculating the attributes of the cluster, removing discrete points, etc. Most of the non-human leg point cloud clusters can be filtered out.

[0212] 7. Kalman filtering is performed on the radar point cloud cluster.

[0213] 8. If the image-locked pedestrian target tracking does not correspond to the radar point cloud cluster tracking, it is considered to be lost, which greatly reduces the probability of being lost. This is mutual verification.

[0214] 9. If the pedestrian on the image is blocked, but the radar point cloud human leg cluster point cloud is not blocked, the last frame of the radar pedestrian tracking can be found near the point cloud to filter the human leg point cloud, so as to continue to lock the pedestrian tracking, and greatly reduce the probability of losing the pedestrian after being blocked.

[0215] 10. When the pedestrian on the image reappears after being blocked, the locked pedestrian can be found again according to the association between the human leg point cloud cluster and the image.

[0216] III. Determining the tracking information of the mobile robot

[0217] (I) Obtaining input information

[0218] Exemplarily, the target object position of the target mobile object is obtained based on the object information. The obstacle information at the current time is obtained, and the mileage position of the mobile robot at the current time is obtained.

[0219] (II) When the tracking target can be detected, the steps of determining the tracking information

[0220] Exemplarily, the distance between the target object position and the mileage position is calculated. If the distance is less than or equal to a first preset distance, a brake instruction is generated; if the distance is greater than the first preset distance, the tracking information of the mobile robot is determined based on the obstacle information, the target object position, and the mileage position.

[0221] For example, if the target is more than 1.5 m away from the robot (the distance value is a set value and can be changed), a moving speed that satisfies the kinematic constraints of the robot and does not collide with the obstacles can be optimized based on the TEB algorithm using the obstacle map and the trajectory of the target walking. If the target is less than 1.5 m away from the robot, the robot is controlled to brake. In this way, the mobile robot can not be too close to the pedestrian, reducing the possibility of danger and ensuring the safety of the tracking process.

[0222] (III) When the tracking target cannot be detected, the steps of determining the tracking information

[0223] Exemplarily, if the target mobile object is not identified from the mobile objects, a disappearance position corresponding to the target mobile object is obtained. If a distance between the disappearance position and the mileage position is greater than or equal to a second preset distance, the second preset distance being smaller than the first preset distance, a moving instruction is generated based on the disappearance position, the moving instruction being used to control the mobile robot to reach the disappearance position, so that the target mobile object can be identified by the mobile robot at the disappearance position.

[0224] If the distance between the disappearance position and the mileage position is smaller than the second preset distance, if the moving direction is opposite to the moving direction of the mobile robot, a waiting instruction is generated to control the mobile robot to wait at the original position for a preset time, and after the waiting preset time, the mobile robot is controlled to rotate at the original position to be able to identify the target mobile object. If the moving direction is the same as the moving direction of the mobile robot, a moving instruction is generated based on the disappearance position, the moving instruction being used to control the mobile robot to reach the disappearance position.

[0225] For example, referring to Figure 12 If the target is lost at a far distance (greater than 1.0m) from the robot, the distance requirement of 1.5m is no longer considered, and the robot is moved to the position where the target disappears and the direction is changed to try to continue to see the pedestrian. If the target is lost at a short distance (less than 1.0m) from the robot, it is determined whether the robot and the pedestrian are moving in opposite directions. If the robot and the pedestrian are moving in opposite directions, the robot is tried to be waited for a certain time (1s, which can be changed to be longer or shorter), so that the safety of the target can be ensured in the case that the target tries to control the robot or places an object on the robot. After the waiting time is exceeded, the robot is controlled to rotate at the original position to the direction when the target is finally lost, and the robot is tried to continue to see the pedestrian to follow the target again.

[0226] Through the above steps, the sensors originally possessed by the robot are used without additional cost, and the following is greatly reduced, and the stability of following is improved. The processing after the pedestrian is lost is increased, and the probability of observing the person again after the loss is improved.

[0227] It should be understood that, although each step in the flowchart involved in each of the above embodiments is shown in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, at least part of the steps in the flowchart involved in each of the above embodiments can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or steps or stages in other steps.

[0228] Based on the same inventive concept, the embodiments of the present application also provide a target following device for implementing the target following method described above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more target following device embodiments provided below can refer to the limitations of the target following method in the above, which will not be repeated here.

[0229] In an exemplary embodiment, as shown in Figure 13 A target following device 1300 is provided, comprising an image detection module 1301, a radar detection module 1302, an identification module 1303, an association module 1304, a determination module 1305, and a following module 1306, wherein:

[0230] The image detection module 1301 is configured to obtain a part detection box of a moving object in a current frame image of a target scene at a current time, the part detection box being used to label a target part of the moving object.

[0231] The radar detection module 1302 is configured to project a point cloud collected by a radar sensor to the current frame image to obtain a projection point set corresponding to the part detection box.

[0232] The identification module 1303 is configured to identify a target moving object from the moving object.

[0233] The association module 1304 is configured to determine a target projection point set corresponding to the target moving object from the projection point set based on a part detection box corresponding to the target moving object.

[0234] The determination module 1305 is configured to obtain object information of the target moving object at the current time based on the target projection point set.

[0235] The following module 1306 is configured to obtain following information of a mobile robot based on the object information, so as to follow the target moving object based on the following information.

[0236] In some embodiments, in terms of determining a target projection point set corresponding to the target moving object from the projection point set based on a part detection box corresponding to the target moving object, the association module 1304 is further configured to: determine a reference point corresponding to the target moving object based on the part detection box corresponding to the target moving object; determine a candidate projection point set located in the part detection box corresponding to the target moving object from the projection point set; and determine the target projection point set from the candidate projection point set based on a distance between the reference point and the candidate projection point set.

[0237] In the determining the target projection point set from the candidate projection point set based on the distance between the reference point and the candidate projection point set, the association module 1304 is further configured to: calculate a center point corresponding to each candidate projection point set; calculate a distance between each center point and the reference point; determine a target distance meeting a first target condition from the distances; and determine the candidate projection point set corresponding to the target distance as the target projection point set.

[0238] In some embodiments, the association module 1304 is further configured to: if a bounding box of a part of a moving object in a current frame image of a current time in a target scene is not obtained, obtain a projection point set based on a point cloud, calculate a center point corresponding to the projection point set, obtain a last frame position of the target moving object at a last time, and determine a target projection point set from the projection point set based on a distance between the last frame position and the center point.

[0239] In some embodiments, in the determining the target projection point set from the projection point set based on the distance between the last frame position and the center point, the association module 1304 is further configured to: predict a position of the target moving object in the current frame image based on the last frame position corresponding to the target moving object, to obtain a predicted position; calculate a first distance between the last frame position and each center point; select a candidate projection point set corresponding to a target distance from the projection point set, the target distance being smaller than other distances in the first distances; calculate a second distance between the center point corresponding to the candidate projection point set and the predicted position; and if the second distance is smaller than or equal to a preset distance, determine the candidate projection point set corresponding to the second distance as the target projection point set.

[0240] In some embodiments, in the projecting the point cloud collected by the radar sensor to the current frame image to obtain the projection point set corresponding to the bounding box, the radar detection module 1302 is further configured to: obtain a mapping relationship between a visual sensor and the radar sensor based on parameters of the visual sensor and parameters of the radar sensor, the visual sensor being configured to capture the target scene; obtain a projection point position corresponding to each point of the point cloud in the current frame image based on the mapping relationship, to obtain a plurality of projection points corresponding to the point cloud; mark the projection points located in the bounding box as object projection points; and perform clustering processing on the object projection points, to obtain the projection point set corresponding to the bounding box.

[0241] In some embodiments, in determining the object information of the target moving object at the current time based on the set of target projection points, the determining module 1305 is further configured to: if the number of the set of target projection points is two, calculate a first center point corresponding to the two sets of target projection points, obtain a first coordinate of a midpoint between the two first center points in the current frame image, convert the first coordinate of the midpoint to the odometry coordinate system to obtain a first global position corresponding to the midpoint, and determine the current position of the target moving object in the target scene based on the first global position; if the number of the set of target projection points is one, calculate a second center point corresponding to the set of target projection points, obtain a second coordinate of the second center point in the current frame image, convert the second coordinate to the odometry coordinate system to obtain a second global position of the second coordinate in the odometry coordinate system, and determine the current position of the target moving object in the target scene based on the second global position; and determine the object information of the target moving object at the current time based on the current position.

[0242] In some embodiments, in determining the object information of the target moving object at the current time based on the current position, the determining module 1305 is further configured to: input the current position to a preset filter; and update the current position of the target moving object at the current time based on the output information of the preset filter.

[0243] In some embodiments, in identifying the target moving object from the moving objects, the identifying module 1303 is further configured to: obtain key point information corresponding to the moving objects in the current frame image; determine the candidate moving objects based on the key point information corresponding to the moving objects; obtain a midpoint of a bottom side of a part detection box corresponding to the candidate moving objects to obtain a coordinate position of the candidate moving objects in the visual sensor coordinate system; convert the coordinate position to the robot coordinate system to obtain a corresponding local coordinate; and determine the candidate moving object closest to the mobile robot as the target moving object based on the local coordinates corresponding to the candidate moving objects.

[0244] In some embodiments, in identifying the target moving object from the moving objects, the identifying module 1303 is further configured to: obtain a last detection box corresponding to the moving objects in a last frame image obtained at a last time; obtain a current detection box corresponding to the moving objects in a current frame image obtained at a current time; obtain odometry information of the mobile robot; convert the current detection box to a converted detection box at the last time based on the odometry information; obtain a correlation relationship between the last detection box in the last frame image and the current detection box in the current frame image based on an overlapping degree between the converted detection box and the last detection box; and determine the target moving object in the current frame image based on the last detection box corresponding to the target moving object in the last frame image and the correlation relationship.

[0245] In some embodiments, in obtaining the odometer information of the mobile robot, the identification module 1303 is further configured to: obtain last odometer information of the mobile robot at a last time and current odometer information of the mobile robot at a current time; in converting the current detection frame to a converted detection frame at the last time based on the odometer information, the identification module is further configured to: obtain a current reference point of the moving object at the current time; based on the last odometer information and the current odometer information, obtain a target transformation matrix, the target transformation matrix being used to represent a transformation relationship of the position information at the current time relative to the position information at the last time; based on the target transformation matrix, transform the position of the current reference point to a coordinate system based on the last time to obtain a transformed position corresponding to the current reference point; obtain size information of the detection frame of the moving object at the current time; and based on the size information and the transformed position, obtain the converted detection frame of the moving object in the last frame of image.

[0246] In some embodiments, in obtaining the following information of the mobile robot based on the object information, the following module 1306 is further configured to: determine a target object position of the target moving object based on the object information; obtain obstacle information at the current time, and obtain a mileage position of the mobile robot at the current time, the mileage position being a position of the mobile robot in an odometer coordinate system, the odometer coordinate system being constructed based on a position where the mobile robot starts to recognize the target moving object as a coordinate origin; calculate a distance between the target object position and the mileage position; if the distance is less than a preset distance, generate a brake instruction; and if the distance is greater than the preset distance, determine the following information of the mobile robot based on the obstacle information, the target object position, and the mileage position.

[0247] In some embodiments, the following module 1306 is further configured to: if the target moving object is not recognized from the moving object, obtain a disappearance position corresponding to the target moving object; if a distance between the disappearance position and the mileage position is greater than or equal to a second preset distance, generate a moving instruction based on the disappearance position, the moving instruction being used to control the mobile robot to reach the disappearance position, so that the mobile robot can recognize the target moving object at the disappearance position, the second preset distance being less than the first preset distance; and if the distance between the disappearance position and the mileage position is less than the second preset distance, obtain a moving direction of the target moving object, and determine the following information of the mobile robot based on the moving direction.

[0248] In some embodiments, in obtaining the moving direction of the target moving object and determining the following information of the mobile robot based on the moving direction, the following module 1306 is further configured to: if the moving direction is opposite to a moving direction of the mobile robot, generate a waiting instruction to control the mobile robot to wait in place for a preset time, and control the mobile robot to rotate in place after the waiting preset time to be able to recognize the target moving object.

[0249] The modules in the target following device can be implemented by software, hardware, or a combination thereof. The modules can be embedded in a processor in a computer device or independent of the processor in hardware, or stored in a memory in the computer device in software, so as to be called and executed by the processor to perform the operations of the modules.

[0250] In an exemplary embodiment, a computer device, which can be a server, is provided. An internal structure diagram of the computer device can be as shown in FIG. 1. Figure 14 The computer device includes a processor, a memory, an input / output interface, and a communication interface. The processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is configured to exchange information between the processor and external devices. The communication interface of the computer device is configured to communicate with external terminals through a network connection. The computer program is executed by the processor to implement a target following method.

[0251] In an exemplary embodiment, a computer device, which can be a robot, is provided. An internal structure diagram of the computer device can be as shown in FIG. 2. Figure 15The computer device shown in the figure includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capability. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals in a wired or wireless manner. The wireless manner can be realized through WIFI, mobile cellular network, near field communication (Near Field Communication, NFC) or other technologies. The computer program is executed by the processor to realize a target following method. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad or mouse, etc.

[0252] Those skilled in the art can understand that, Figure 14 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0253] In one exemplary embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps in each of the above method embodiments.

[0254] In one embodiment, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the steps in each of the above method embodiments.

[0255] In one embodiment, a computer program product is provided, including a computer program, and the computer program is executed by a processor to implement the steps in each of the above method embodiments.

[0256] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0257] It can be understood by those skilled in the art that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing related hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments of each method. In the embodiments provided in the present application, any reference to memory, database or other medium can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (Resistive Random Access Memory, ReRAM), magnetoresistive random access memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric memory (Ferroelectric Random Access Memory, FRAM), phase change memory (Phase Change Memory, PCM), graphene memory, etc. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (Artificial Intelligence, AI) processor, etc., without being limited thereto.

[0258] Any technical features in the above embodiments can be combined, and for the sake of brevity, not all possible combinations are described above, however, any combination of these technical features is deemed to be within the scope of the present application.

[0259] The above embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation on the patent scope of the present application. It should be pointed out that, for ordinary skilled persons in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A target following method applied to a mobile robot, characterized by, The method comprises: obtaining a part detection frame of a moving object in a target scene in a current frame image at a current time, the part detection frame being used for labeling a target part of the moving object; projecting a point cloud collected by a radar sensor to the current frame image to obtain a projection point set corresponding to the part detection frame; identifying a target moving object from the moving object; determining a target projection point set corresponding to the target moving object from the projection point set based on a part detection frame corresponding to the target moving object; obtaining object information of the target moving object at the current time based on the target projection point set; obtaining following information of the mobile robot based on the object information, so as to follow the target moving object based on the following information.

2. The method of claim 1, wherein, The method further comprises: determining a reference point corresponding to the target moving object based on the part detection frame corresponding to the target moving object; determining a candidate projection point set located in the part detection frame corresponding to the target moving object from the projection point set; determining the target projection point set from the candidate projection point set based on distances between the reference point and the candidate projection point set.

3. The method of claim 2, wherein, The method further comprises: calculating a center point corresponding to each of the candidate projection point sets; calculating distances between each of the center points and the reference point; determining a target distance from each of the distances that meets a first target condition; determining the candidate projection point set corresponding to the target distance in the candidate projection point set as the target projection point set.

4. The method of claim 1, wherein, The method further comprises: if no part detection frame of a moving object in a target scene in a current frame image at a current time is obtained, determining a projection point set based on the point cloud, calculating a center point corresponding to the projection point set, obtaining a last frame position of the target moving object at a last time, and determining the target projection point set from the projection point set based on a distance between the last frame position and the center point.

5. The method of claim 4, wherein, The method further comprises: predicting a position of the target moving object at the current time based on the last frame position corresponding to the target moving object to obtain a predicted position; calculating a first distance between the last frame position and each of the center points; selecting a candidate projection point set corresponding to a target distance from the projection point set, the target distance being smaller than other distances in the first distances; calculating a second distance between a center point corresponding to the candidate projection point set and the predicted position; if the second distance is smaller than or equal to a preset distance, determining the candidate projection point set corresponding to the second distance as the target projection point set.

6. The method of claim 1, wherein, The projecting the point cloud collected by the radar sensor to the current frame image to obtain a projection point set corresponding to the part detection box comprises: obtaining a mapping relationship between the visual sensor and the radar sensor based on parameters of the visual sensor and parameters of the radar sensor, the visual sensor being configured to capture the target scene; obtaining a projection point position corresponding to each point of the point cloud in the current frame image based on the mapping relationship, to obtain a plurality of projection points corresponding to the point cloud; marking the projection points located in the part detection box as object projection points; performing clustering processing on the object projection points to obtain the projection point set corresponding to the part detection box.

7. The method of claim 2 or 4, wherein, The obtaining object information of the target moving object at the current time based on the target projection point set comprises: if the number of the target projection point set is two, calculating a first center point corresponding to the two target projection point sets, obtaining a first coordinate of a midpoint between the two first center points in the current frame image, converting the first coordinate of the midpoint to an odometer coordinate system to obtain a first global position corresponding to the midpoint, and determining a current position of the target moving object in the target scene based on the first global position; if the number of the target projection point set is one, calculating a second center point corresponding to the target projection point set, obtaining a second coordinate of the second center point in the current frame image, converting the second coordinate to the odometer coordinate system to obtain a second global position of the second coordinate in the odometer coordinate system, and determining a current position of the target moving object in the target scene based on the second global position; determining the object information of the target moving object at the current time based on the current position.

8. The method of claim 7, wherein, The determining the object information of the target moving object at the current time based on the current position comprises: inputting the current position to a preset filter; updating the current position of the target moving object at the current time based on output information of the preset filter.

9. The method of claim 1, wherein, The identifying a target moving object from the moving object comprises: obtaining key point information corresponding to the moving object in the current frame image; determining a candidate moving object based on the key point information corresponding to the moving object; obtaining a midpoint of a bottom side of the part detection box corresponding to the candidate moving object to obtain a coordinate position of the candidate moving object in a visual sensor coordinate system; converting the coordinate position to a robot coordinate system to obtain a corresponding local coordinate; determining a candidate moving object closest to the mobile robot as the target moving object based on the local coordinate corresponding to each candidate moving object.

10. The method of claim 1, wherein, The identifying a target moving object from the moving object comprises: obtaining a last detection box corresponding to the moving object in a last frame image obtained at a last time; obtaining a current detection box corresponding to the moving object in the current frame image; obtaining odometer information of the mobile robot; convert the current detection frame to a converted detection frame at the previous time point based on the odometer information; obtain a correlation relationship between the previous detection frame in the previous frame of image and the current detection frame in the current frame of image based on a degree of coincidence between the converted detection frame and the previous detection frame; determine the target moving object in the current frame of image based on the previous detection frame corresponding to the target moving object in the previous frame of image and the correlation relationship.

11. The method of claim 10, wherein, The method further comprises: obtain the odometer information of the mobile robot, including: obtain previous odometer information of the mobile robot at the previous time point and current odometer information of the mobile robot at the current time point; convert the current detection frame to a converted detection frame at the previous time point based on the odometer information, including: obtain a current reference point of the moving object at the current time point; obtain a target transformation matrix based on the previous odometer information and the current odometer information, the target transformation matrix being used to represent a transformation relationship of image coordinate information at the current time point relative to image coordinate information at the previous time point; map the current reference point to a transformed reference point based on the previous time point coordinate system based on the target transformation matrix; obtain size information of a detection frame of the moving object at the current time point; 12. The method of claim 1, wherein, obtain the converted detection frame of the moving object in the previous frame of image based on the size information and the transformed reference point. The method further comprises: obtain a target object position of the target moving object based on the object information; obtain obstacle information at the current time point, obtain a mileage position of the mobile robot at the current time point, the mileage position being a position of the mobile robot in an odometer coordinate system, the odometer coordinate system being constructed with a position as a coordinate origin, the position being started to be activated after the mobile robot identifies the target moving object; calculate a distance between the target object position and the mileage position; if the distance is less than or equal to a first preset distance, generate a brake instruction; 13. The method of claim 12, wherein, if the distance is greater than the first preset distance, determine the following information of the mobile robot based on the obstacle information, the target object position and the mileage position. The method further comprises: if the target moving object cannot be identified from the moving object, obtain a disappearance position corresponding to the target moving object; if a distance between the disappearance position and the mileage position is greater than or equal to a second preset distance, generate a moving instruction based on the disappearance position, the moving instruction being used to control the mobile robot to reach the disappearance position, so that the mobile robot can identify the target moving object at the disappearance position, the second preset distance being less than the first preset distance. If a distance between the disappearance position and the mileage position is less than the second preset distance, a moving direction of the target moving object is obtained, and following information of the mobile robot is determined based on the moving direction.

14. The method of claim 13, wherein, The obtaining of the moving direction of the target moving object and the determining of the following information of the mobile robot based on the moving direction comprise: If the moving direction is opposite to a moving direction of the mobile robot, a waiting instruction is generated to control the mobile robot to wait at a same position for a preset time, and after waiting for the preset time, the mobile robot is controlled to rotate at the same position to be able to recognize the target moving object.

15. A target following device, characterized by The apparatus comprises: An image detection module is configured to obtain a part detection frame of a moving object in a current frame image of a target scene at a current time, the part detection frame being used to mark a target part of the moving object; A radar detection module is configured to project point clouds collected by a radar sensor to the current frame image to obtain a projection point set corresponding to the part detection frame; An identification module is configured to identify a target moving object from the moving object; An association module is configured to determine a target projection point set corresponding to the target moving object from the projection point set based on a part detection frame corresponding to the target moving object; A determination module is configured to obtain object information of the target moving object at the current time based on the target projection point set; A following module is configured to obtain following information of a mobile robot based on the object information, and to follow the target moving object based on the following information. 16.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-15. The processor executes the computer program to implement the steps of the method in any one of claims 1 to 14.

17. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 14.

18. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 14.

Images