Robot target tracking method and device, robot, and storage medium

Abstract

The application relates to a robot target tracking method and device, a robot and a storage medium, the method comprising the following steps: target detection is performed on a first image and a second image to obtain at least one first object and at least one second object; the first image is collected based on a first camera, and the second image is collected based on a second camera; the field of view range of the second camera is larger than that of the first camera, and the field of view ranges of the first camera and the second camera overlap; image features and position features of each first object and second object in the first image and the second image are extracted; target objects in the overlapping field of view range are determined based on the image features and the position features; third image features are obtained by fusing the image features corresponding to the target objects; and the target objects are tracked based on the third image features, so that the tracking field of view of the target objects is expanded, and the problem that robots are difficult to track the overall movement process of a tracked target in the prior art is solved.

Description

Technical Field

[0001] This application relates to the field of robot vision tracking technology, and in particular to a robot target tracking method, device, robot, and readable storage medium. Background Technology

[0002] Intelligent robots can perceive their surroundings through cameras mounted on their bodies, and multi-object tracking within the video images captured by these cameras is a crucial prerequisite for robots to understand their environment. For example, multi-object tracking technology can provide robots with real-time information about pedestrians, vehicles, and other obstacles, thus providing a basis for intelligent obstacle avoidance. Multi-object tracking also helps robots select service targets, enabling them to better serve humans. Existing multi-object tracking technologies are mostly based on the robot's own field of view. Due to the close proximity to the tracked target, detailed features can be obtained. However, due to limitations in robot hardware, the number of cameras mounted on the robot is usually small, the installation height is low, and the field of view is limited, making it difficult to track the overall movement of the target. Therefore, achieving overall movement tracking of the tracked target while preserving detailed features is a pressing problem that needs to be solved.

[0003] There is currently no effective solution to the problem that robots in related technologies cannot track the overall motion of the target being tracked. Summary of the Invention

[0004] This embodiment provides a robot target tracking method, device, robot, and readable storage medium to solve the problem in related technologies that robots have difficulty tracking the overall motion process of the tracked target.

[0005] Firstly, this embodiment provides a robot target tracking method, the method comprising:

[0006] Perform object detection on the first image to obtain at least one first object; and

[0007] Target detection is performed on the second image to obtain at least one second object; the first image is acquired based on the first camera, the second image is acquired based on the second camera, the field of view of the second camera is larger than the field of view of the first camera, and there is an overlap between the field of view of the second camera;

[0008] Extract the first image features and first position features of each first object in the first image, and the second image features and second position features of each second object in the second image;

[0009] Based on the first image features and the first location features, as well as the second image features and the second location features, the target object in the overlapping field of view of the first object and the second object is determined;

[0010] The first image feature and the second image feature corresponding to the target object are fused to obtain the third image feature, and the target object is tracked based on the third image feature.

[0011] In some embodiments, determining the target object within the overlapping field of view of the first object and the second object based on the first image features and the first location features, as well as the second image features and the second location features, includes:

[0012] Obtain the image feature distance between each of the first image features and each of the second image features; and

[0013] Obtain the position feature distance between each of the first position features and each of the second position features;

[0014] Based on the image feature distances and location feature distances, the target object in the overlapping field of view between the first object and the second object is determined.

[0015] In some embodiments, determining the target object within the overlapping field of view of the first object and the second object based on the respective image feature distances and position feature distances includes:

[0016] Based on the distances of the aforementioned image features, an image distance matrix is ​​constructed; and

[0017] Based on the distances of the aforementioned location features, a location distance matrix is ​​constructed;

[0018] The image distance matrix and the location distance matrix are combined to obtain a distance information matrix;

[0019] Using the value of the distance information matrix as a target object determination parameter, the target object that is within the overlapping field of view between the first object and the second object is determined.

[0020] In some embodiments, the first position feature is the three-dimensional coordinates of the corresponding first object in the world coordinate system; the first position feature is determined based on the transformation relationship between the robot coordinate system and the world coordinate system at the current moment, the transformation relationship between the first camera coordinate system and the robot coordinate system, and the homogeneous three-dimensional coordinates of the corresponding first object in the first camera coordinate system.

[0021] In some embodiments, the second position feature is the three-dimensional coordinates of the corresponding second object in the world coordinate system; the second position feature is determined based on the transformation relationship between the second camera and the world coordinate system and the homogeneous three-dimensional coordinates of the corresponding second object in the second camera coordinate system.

[0022] In some embodiments, fusing the first image feature and the second image feature corresponding to the target object to obtain a third image feature, and tracking the target object based on the third image feature, includes:

[0023] The first image feature and the second image feature corresponding to the target object are concatenated to obtain the third image feature.

[0024] In some embodiments, concatenating the first image feature and the second image feature corresponding to the target object to obtain the third image feature includes:

[0025] F = n([F i ;F j ])

[0026] Where F is the third image feature; n() is the normalization function, F i F represents the first image feature corresponding to the i-th first object. j This refers to the second image feature corresponding to the j-th second object.

[0027] Secondly, this embodiment provides a robot target tracking device, the robot target tracking device comprising:

[0028] The first detection module is used to perform target detection on the first image to obtain at least one first object;

[0029] The second detection module is used to perform target detection on the second image to obtain at least one second object; the first image is acquired based on the first camera, the second image is acquired based on the second camera, the field of view of the second camera is larger than the field of view of the first camera, and there is an overlap between the field of view of the second camera and the field of view of the first camera;

[0030] An extraction module is used to extract the first image features and the first position features of each first object in the first image, and the second image features and the second position features of each second object in the second image;

[0031] The determination module is used to determine, based on the first image features and the first position features, as well as the second image features and the second position features, the target object in the overlapping field of view of the first object and the second object;

[0032] The tracking module is used to fuse the first image feature and the second image feature corresponding to the target object to obtain a third image feature, and to track the target object based on the third image feature.

[0033] Thirdly, this embodiment provides a robot, the robot comprising:

[0034] A first camera for acquiring a first image, and a robot target tracking device as described in the second aspect.

[0035] Fourthly, this embodiment provides a readable storage medium on which a program is stored, which, when executed by a processor, implements the steps of the robot target tracking method described in the first aspect above.

[0036] Compared with related technologies, the robot target tracking method provided in this embodiment obtains at least one first object by performing target detection on a first image and at least one second object by performing target detection on a second image. The first image is captured by a first camera, and the second image is captured by a second camera. The field of view of the second camera is larger than that of the first camera, and they overlap. That is, target detection is performed separately to obtain the objects to be matched from the two images captured by the two cameras. The field of view of the first camera is smaller, while that of the second camera is larger, enabling the acquisition of the overall motion process of the objects to be matched. The method further extracts the first image features and first position features of each first object in the first image, and the first image features of each second object in the second image. The image features and positional features of the target object are obtained by using the second image features and the second positional features in the image. Based on the first image features and the first positional features, as well as the second image features and the second positional features, the target object in the overlapping field of view of the first object and the second object is determined, that is, the target object in the target object is determined, which is in both the first object and the second object. By fusing the first image features and the second image features corresponding to the target object, a third image feature is obtained. The target object is tracked based on the third image feature to obtain the fused image feature of the target object. This expands the tracking field of view of the target object, solves the problem in related technologies that robots have difficulty tracking the overall motion process of the tracked target, and improves the tracking effect.

[0037] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0038] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0039] Figure 1 This is a schematic diagram illustrating the application environment of some embodiments of the robot target tracking method of this application;

[0040] Figure 2 This is a flowchart of a robot target tracking method according to some embodiments of this application;

[0041] Figure 3 This is a flowchart illustrating the determination of target objects based on image features and location features according to some embodiments of this application;

[0042] Figure 4 This is a flowchart illustrating the determination of target objects based on image feature distance and location feature distance according to some embodiments of this application;

[0043] Figure 5 This is a flowchart of a robot target tracking method according to some preferred embodiments of this application;

[0044] Figure 6 This is a structural block diagram of a robot target tracking device according to some embodiments of this application. Detailed Implementation

[0045] To better understand the purpose, technical solution, and advantages of this application, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0046] Unless otherwise defined, the technical or scientific terms used in this application shall have the general meaning as understood by one of ordinary skill in the art to which this application pertains. Words such as “a,” “an,” “an,” “the,” “the,” and “these,” used in this application, do not indicate quantitative limitation and may be singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include steps or modules (units) not listed, or may include other steps or modules (units) inherent to such processes, methods, products, or devices. The terms “connected,” “linked,” and “coupled,” used in this application, are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. The term “multiple” used in this application refers to two or more. The "and / or" operator describes the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: A alone, A and B simultaneously, and B alone. Typically, the character " / " indicates that the objects before and after it are in an "or" relationship. The terms "first," "second," and "third," etc., used in this application are merely for distinguishing similar objects and do not represent a specific ordering of the objects.

[0047] The robot target tracking method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, robot 10 can communicate with second camera 12 via a network, and second camera 12 can send the acquired image data to robot 10. Robot 10 includes a first camera 100, and the image data acquired by the first camera 100 can be stored in the storage system of robot 10. The image data acquired by the first camera 100 and the second camera 12 can also be stored on the cloud or other network servers via a network. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the robot described above. For example, the robot may also include components that are larger than... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown are illustrated.

[0048] The robot target tracking method of this application embodiment can be applied to the control unit of robot 10, or to a computer that communicates with robot 10 and second camera 12 via a network. The computer can be, but is not limited to, various servers, personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. Portable wearable devices can be smartwatches, smart bracelets, head-mounted devices, etc. Taking the control unit of robot 10 as an example, the control unit can include one or more processors and a memory for storing data. The processor can be, but is not limited to, a microprocessor (MCU) or a programmable logic device (FPGA). The control unit can also include transmission devices for communication functions and input / output devices, enabling communication with a remote server via a network and data processing and storage through the remote server.

[0049] This embodiment provides a robot target tracking method. Figure 2 This is a flowchart of a robot target tracking method according to some embodiments of this application, such as... Figure 2 As shown, the process includes the following steps:

[0050] Step S201: Perform target detection on the first image to obtain at least one first object.

[0051] The first camera can be mounted on the robot body to monitor the robot's surrounding environment, vehicles, pedestrians, etc., and to acquire images for target detection and analysis as needed. There can be one or more first cameras, and correspondingly, the first image can be one or more images; the first image can be a color image or an RGBD image, i.e., a color image with depth information. Target detection refers to using target detection algorithms to detect targets in the image, where the targets can be pedestrians, vehicles, etc. There can be one or more first objects. Preferably, the detected first objects can be identified using bounding rectangles; preferably, deep neural network-based target detection algorithms, such as YOLOv3 and Mask R-CNN, can be used.

[0052] Step S202: Target detection is performed on the second image to obtain at least one second object. The first image is acquired based on the first camera, and the second image is acquired based on the second camera. The field of view of the second camera is larger than that of the first camera, and they overlap.

[0053] The second camera can be placed in the environment or other equipment surrounding the robot as a monitoring camera. The type and placement of the second camera should ensure that its field of view is larger than that of the first camera and overlaps with it. For example, the first camera can be placed at the front of the robot, and the second camera can be placed above the road where the robot is located. Similarly, there can be one or more second cameras, and the second image can be one or more images; the second image can be a color image or an RGBD image. There can be one or more second objects. It should be noted that the first and second images should be synchronized to ensure that all images are acquired at the same time.

[0054] Steps S201 and S202 can be interchanged.

[0055] Step S203: Extract the first image features and first position features of each first object in the first image, and the second image features and second position features of each second object in the second image.

[0056] Image features can be characteristics of pedestrians' clothing and behavior, or vehicle colors, license plate numbers, etc., extracted from images to identify targets. Image features can be obtained using deep learning models or feature extraction algorithms to capture the image features of the first and second objects in an RGB or RGBD image. Positional features can be the location of the pedestrian or vehicle, which can be a real-world location or a relative position based on the same camera coordinate system or robot coordinate system. Since the first and second images were captured by different cameras, and there can be multiple first or second cameras, the positional features of different objects can be transformed to obtain their relative positions in the same coordinate system.

[0057] Step S204: Based on the first image features and the first position features, as well as the second image features and the second position features, determine the target object in the overlapping field of view between the first object and the second object.

[0058] Because the fields of view of the first and second cameras overlap, the same object may be present in the fields of view of both cameras simultaneously; such an object is called a target object. Target detection is performed on the first and second images to identify all objects of interest, both first and second objects. Then, image features and positional features are used to match the first and second objects to determine potential target objects. For example, the first image may include one image captured by the first camera, and the second image may include one image each captured by two second cameras (e.g., surveillance camera A and surveillance camera B). If two first objects are detected in the first image, two second objects in surveillance camera A, and three second objects in surveillance camera B, then image features and positional features are extracted from the three images to obtain a single target object that appears in all three images.

[0059] Step S205: The first image feature and the second image feature corresponding to the target object are fused to obtain the third image feature, and the target object is tracked based on the third image feature.

[0060] If the target object exists, that is, the target object corresponds to both the first object and the second object, and has the image features of both the first object and the second object, that is, it contains both the detailed features of the tracked target and a wide field of view, the two image features can be fused to obtain a more complete third image feature, and the third image feature can be used for joint field of view tracking.

[0061] For other first and second objects besides the target object, i.e., first objects that only appear in the first image and not in the second image, and second objects that only appear in the second image and not in the first image, these first and second objects can be tracked based on the original image features within their respective fields of view.

[0062] Specifically, multi-target tracking can use trackers with a tracking-by-detection framework, preferably trackers such as DeepSORT.

[0063] Through the above steps S201 to S205, at least one first object is obtained by performing target detection on the first image, and at least one second object is obtained by performing target detection on the second image. The first image is captured by the first camera, and the second image is captured by the second camera. The field of view of the second camera is larger than that of the first camera, and they overlap. That is, target detection is performed on the two images captured by the two cameras to obtain the objects to be matched. The field of view of the first camera is smaller, and that of the second camera is larger, enabling the acquisition of the overall motion process of the objects to be matched. The first image features and first position features of each first object in the first image, and the second image features of each second object in the second image are extracted. Image features and second position features are used to obtain the image features and position features of the object to be matched. Based on the first image features and first position features, as well as the second image features and second position features, the target object in the overlapping field of view of the first object and the second object is determined, that is, the target object existing in the object to be matched is determined, which is in both the first object and the second object. By fusing the first image features and second image features corresponding to the target object, a third image feature is obtained. The target object is tracked based on the third image feature to obtain the fused image feature of the target object, which expands the tracking field of view of the target object and solves the problem in related technologies that robots have difficulty tracking the overall motion process of the tracked target, thus improving the tracking effect.

[0064] In some of these embodiments, Figure 3 This is a flowchart illustrating the determination of target objects based on image features and location features according to some embodiments of this application. For example... Figure 3 As shown, the process includes the following steps:

[0065] Step S301: Obtain the image feature distance between each first image feature and each second image feature.

[0066] In the image feature extraction process, image feature vectors can be used to represent the image features of a first object and a second object in the image. Specifically, a feature extractor can be used to extract the image features of the first object or the second object. Image features extracted from the same object at different times or from different viewpoints are close in distance or have high similarity, while image features extracted from different objects are far apart or have low similarity. Preferably, in application scenarios where pedestrians are the tracked targets, a pedestrian re-identification model can be used as the feature extractor.

[0067] The image feature distance between each first object and each second object can be obtained by calculating the similarity between image feature vectors, such as Euclidean distance or cosine similarity. Preferably, cosine similarity is used to represent the image feature distance, and the calculation formula can be:

[0068] F ij (·)=s(1-n(F i )·n(F j ))

[0069] Among them, F ij (·) represents the image feature distance between the i-th first object in the target detection result of the first image and the j-th second object in the target detection result of the second image; s() represents the SIGMOID function, which maps the image feature distance value to the interval [0,1]; n() represents the normalization function, which ensures that the magnitude of the feature vector is 1; F i F represents the image feature vector of the i-th first object in the target detection result of the first image. j This represents the image feature vector of the j-th second object in the target detection result of the second image.

[0070] Step S302: Obtain the position feature distance between each first position feature and each second position feature.

[0071] The first position feature and the second position feature can be the position coordinates of the first object and the second object based on the same coordinate system. This coordinate system can be the world coordinate system or other coordinate systems, such as the camera coordinate system of the first camera or the second camera.

[0072] After obtaining the first and second positional features, the positional feature distance between the first and second objects can be obtained by calculating the distance between two points in the real-world coordinate system or other coordinate systems. Alternatively, the positional feature distance can be calculated using Euclidean distance, cosine distance, etc. Preferably, in the world coordinate system, Euclidean distance is used to represent the image feature distance, and the calculation formula can be:

[0073]

[0074] Among them, G ij (·) represents the positional feature distance between the i-th first object in the target detection result of the first image and the j-th second object in the target detection result of the second image; s() represents the SIGMOID function, which maps the distance value to the interval [0,1]; (p ix p iy p iz (p) represents the real-world 3D coordinates of the i-th first object in the target detection result of the first image. jx p jy p jz ) represents the real-world 3D coordinates of the j-th second object in the target detection result of the second image.

[0075] Steps S301 and S302 can be interchanged.

[0076] Step S303: Based on the distances of each image feature and the distances of each location feature, determine the target object in the overlapping field of view between the first object and the second object.

[0077] Image feature distance and location feature distance are two parameters used to characterize the similarity between a first object and a second object. A larger feature distance indicates a lower similarity between the first and second objects, while a smaller feature distance indicates a higher similarity. Based on the image feature distance and location feature distance between each first object and each second object, target objects can be determined in different ways. For example, this can be done by setting separate thresholds for image feature distance and location feature distance, or by combining the values ​​of both feature distances using a matching algorithm for a comprehensive determination.

[0078] Through the above steps S301 to S303, by obtaining the image feature distance between each first image feature and each second image feature, a reference value for determining the target object is obtained from the image similarity between the first object and the second object; by obtaining the position feature distance between each first position feature and each second position feature, a reference value for determining the target object is obtained from the position similarity between the first object and the second object; by determining the target object in the overlapping field of view between the first object and the second object based on each image feature distance and each position feature distance, the target object in the first object and the second object is determined. By determining the target object in the first object and the second object through two reference values, the matching of tracking targets in different tracking fields of view is realized, and the tracking field of view of the target object is expanded.

[0079] In some of these embodiments, Figure 4 This is a flowchart illustrating the determination of target objects based on image feature distance and location feature distance according to some embodiments of this application. For example... Figure 4 As shown, the process includes the following steps:

[0080] Step S401: Construct an image distance matrix based on the distances of each image feature.

[0081] When there are multiple first objects in a first image, or multiple second objects in a second image, the feature distances between the multiple first objects and the multiple second objects can be described using a matrix. For example, an image distance matrix can be constructed where the number of rows is equal to the number of first objects M, and the number of columns is equal to the number of second objects N. Each element in the image distance matrix is ​​equal to the image feature distance between the first object in that row and the second object in that column.

[0082] Step S402: Construct a location distance matrix based on the feature distances of each location.

[0083] Similar to step S401, a positional distance matrix is ​​constructed, wherein the number of rows in the positional distance matrix is ​​equal to the number of first objects M, and the number of columns is equal to the number of second objects N. Each element value in the positional distance matrix is ​​equal to the positional feature distance between the first object in that row and the second object in that column.

[0084] Steps S401 and S402 can be interchanged.

[0085] Step S403: Combine the image distance matrix and the location distance matrix to obtain the distance information matrix.

[0086] The distance information matrix can be obtained by calculating the Hadamard product between the image distance matrix and the location distance matrix. For two M*N matrices of the same order, A and B, the element values ​​of their Hadamard product matrix are equal to the product of the element values ​​of the same row and column of matrix A and the element values ​​of the same row and column of matrix B.

[0087] Step S404: Using the value of the distance information matrix as a target object determination parameter, determine the target object in the overlapping field of view between the first object and the second object.

[0088] The element values ​​of the distance information matrix are the products of the image feature distance and the positional feature distance between the first object and the second object. Combining these two parameters, it can represent the similarity between the first and second objects and serve as a parameter for determining the target object. A similarity threshold can be preset, and the element values ​​of the distance information matrix can be compared with the threshold to determine the target object within the overlapping visual field of the first and second objects. Preferably, the element values ​​of the distance information matrix can be used as weights in a bipartite graph formed by the first and second objects, and the optimal match of the bipartite graph can be solved to obtain the target object within the overlapping visual field of the first and second objects. Preferably, the optimal match of the bipartite graph can be solved using the Hungarian matching algorithm.

[0089] Through the above steps S401 to S404, an image distance matrix is ​​constructed based on the distance of each image feature, and the image distance matrix is ​​used to describe the image feature distance between each first object and each second object; a position distance matrix is ​​constructed based on the distance of each position feature, and the position distance matrix is ​​used to describe the position feature distance between each first object and each second object; a distance information matrix is ​​obtained by combining the image distance matrix and the position distance matrix, and the distance information matrix is ​​used to describe the feature comprehensive distance between each first object and each second object; by using the value of the distance information matrix as a target object determination parameter, the target object in the overlapping field of view of the first object and the second object is determined, thus realizing the matching of multiple tracking targets in different tracking fields of view, expanding the tracking field of view of the target object, and improving the tracking accuracy.

[0090] In some embodiments, a method for obtaining a first position feature of a first object is involved. The first position feature is the three-dimensional coordinates of the corresponding first object in the world coordinate system; the method for obtaining the first position feature includes:

[0091] The first position feature is determined based on the transformation relationship between the robot coordinate system and the world coordinate system at the current moment, the transformation relationship between the first camera coordinate system and the robot coordinate system, and the homogeneous three-dimensional coordinates of the corresponding first object in the first camera coordinate system.

[0092] In this embodiment, the first object refers to the first object corresponding to the target object. The first image is acquired by the first camera on the robot body. The first position feature of the first object can be the three-dimensional coordinates of the first object in the world coordinate system. The first position feature can be calculated by the following formula:

[0093]

[0094] in, This is the transformation matrix between the robot's coordinate system and the world coordinate system at the current moment; This is the transformation matrix between the robot's camera coordinate system and the robot's coordinate system; The homogeneous 3D coordinates of the first object in the camera coordinate system; Let be the homogeneous three-dimensional coordinates of the first object in the world coordinate system.

[0095] Preferably, the world coordinate system can be the coordinate system of the map used for robot positioning and navigation.

[0096] Preferably, for robots using LiDAR for positioning and navigation, the robot coordinate system can be the LiDAR coordinate system; for robots using cameras for positioning and navigation, the robot coordinate system can be the robot camera coordinate system. It is an identity matrix.

[0097] Preferably, the transformation matrix between the robot coordinate system and the world coordinate system, i.e. Provided by the robot's localization module; the transformation matrix between the robot's camera coordinate system and the robot coordinate system, i.e. Provided by the robot's sensor calibration module.

[0098] Preferably, for robots using RGBD cameras, The depth can be directly obtained from the detection results; for robots using RGB cameras, the RGB camera can be calibrated against the ground plane, and the result can be calculated based on the contact point between the target and the ground plane in the target detection results and the calibration results.

[0099] The robot target tracking method in this embodiment transforms the world coordinate system, robot coordinate system, and first camera coordinate system into homogeneous three-dimensional coordinates of the first object in the first camera coordinate system, thereby determining a unified coordinate system for subsequent calculation of the position feature distance between the first object and the second object.

[0100] In some embodiments, a method for obtaining a second position feature of a second object is involved. The second position feature is the three-dimensional coordinates of the corresponding second object in the world coordinate system; the method for obtaining the second position feature includes:

[0101] The second position feature is determined based on the transformation relationship between the second camera and the world coordinate system, as well as the homogeneous three-dimensional coordinates of the corresponding second object in the second camera coordinate system.

[0102] In this embodiment, the second object refers to the second object corresponding to the target object. The second image is acquired by the second camera, which is a monitoring camera in the area where the robot is located. The second positional feature of the second object can be the three-dimensional coordinates of the second object in the world coordinate system, which can be calculated by the following formula:

[0103]

[0104] in, This is the transformation matrix between the second camera and the world coordinate system; The homogeneous 3D coordinates of the second object in the second camera coordinate system; This represents the homogeneous three-dimensional coordinates of the second object in the world coordinate system.

[0105] Preferably, the transformation matrix between the second camera and the real-world coordinate system, i.e. This can be obtained through the calibration of the second camera with the ground plane and physical measurements.

[0106] Preferably, for applications using multiple second cameras, the extrinsic parameters of the multiple second cameras can be calibrated first, and then the triangulation method can be used to obtain the results. For applications using only a single second camera, the second camera can be calibrated against the ground plane first. The contact point between the target and the ground plane in the target detection results, along with the calibration results, can then be used to calculate...

[0107] The robot target tracking method in this embodiment transforms the homogeneous three-dimensional coordinates of the second object in the second camera coordinate system into homogeneous three-dimensional coordinates in the world coordinate system through coordinate system transformation between the world coordinate system and the second camera coordinate system. This provides a unified coordinate system for subsequent calculation of the position feature distance between the first object and the second object.

[0108] In some embodiments, a method for fusing first image features and second image features is also involved. This method includes:

[0109] The first image feature and the second image feature corresponding to the target object are concatenated to obtain the third image feature.

[0110] If a target object exists in both the first and second objects, assuming that the target object corresponds to the i-th first object in the first image target detection result and the j-th second object in the second image target detection result, their image feature vectors are Fi and Fj, respectively. i and F j Then F can be used. i and F j The concatenated image features are used as the third image feature of the target object, and this third image feature is used for multi-target tracking.

[0111] The series formula is:

[0112] F = n([F i ;F j ])

[0113] Where F represents the image features of the target object; n() represents the normalization function, ensuring that the magnitude of the feature vector is 1.

[0114] The robot target tracking method in this embodiment obtains a third image feature by concatenating the first image feature and the second image feature corresponding to the target object. This enhances the accuracy of target detection and recognition in the two fields of view of the first and second cameras, expands the tracking field of view of the target object, and realizes the tracking of the overall motion process of the target object.

[0115] The present embodiment will now be described and illustrated through preferred embodiments.

[0116] Figure 5 This is a flowchart of a robot target tracking method according to some preferred embodiments of this application. For example... Figure 5 As shown, the process includes the following steps:

[0117] Step S501: Obtain the robot body camera image as the first image and the monitoring camera image as the second image. The first and second images should be synchronized in time.

[0118] Step S502: Perform target detection on the first and second images respectively to obtain at least one first object and at least one second object;

[0119] Step S503: Calculate the three-dimensional coordinates of the first object in the world coordinate system to obtain the three-dimensional coordinates of the first object;

[0120] Step S504: Calculate the three-dimensional coordinates of the second object in the world coordinate system to obtain the three-dimensional coordinates of the second object;

[0121] Step S505: Calculate the distance between the three-dimensional coordinates of the first and second objects to obtain the position distance matrix;

[0122] Step S506: Use a feature extractor to extract image features of the first and second objects respectively, to obtain image features of the first object and image features of the second object;

[0123] Step S507: Calculate the distance between the image features of the first and second objects to obtain the image distance matrix;

[0124] The order of steps S503-S505 and S506-S507 can be interchanged.

[0125] Step S508: Combine the location distance matrix and the image distance matrix into a distance information matrix;

[0126] Step S509: Construct a bipartite graph between the first and second objects. The weights of the edges in the bipartite graph are the element values ​​of the distance information matrix. Solve for the best matching of the bipartite graph to obtain the target object.

[0127] Step S510: Combine the image features of the first object and the image features of the second object corresponding to the target object into a third image feature;

[0128] Step S511: Use the third image features to perform multi-target tracking.

[0129] Through steps S501 to S511, target detection is performed to obtain a first object and a second object from two images captured by two cameras as objects to be matched. The first camera has a smaller field of view, while the second camera has a larger field of view, enabling it to capture the overall motion process of the objects to be matched. By extracting image features and positional features of the first and second objects, feature parameter values ​​for matching are obtained. By constructing corresponding matrices based on image feature distance and positional feature distance, the similarity between multiple first objects and multiple second objects is determined. By combining the two matrices into a distance information matrix and solving it using a bipartite graph, the target object within the overlapping field of view of the first and second objects is identified. By fusing the first and second image features corresponding to the target object, a third image feature is obtained. The target object is tracked based on the third image feature to obtain the fused image feature of the target object, which expands the tracking field of view of the target object and solves the problem in related technologies where robots have difficulty tracking the overall motion process of the tracked target, thus improving the tracking accuracy.

[0130] It should be noted that the steps shown in the above process or in the flowchart of the accompanying figures can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0131] In some embodiments, this application also provides a robot target tracking device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. The terms "module," "unit," "subunit," etc., used below refer to combinations of software and / or hardware that perform predetermined functions.

[0132] In some embodiments, Figure 6 This is a structural block diagram of a robot target tracking device according to some embodiments of this application, such as... Figure 6 As shown, the device includes:

[0133] The first detection module 61 is used to perform target detection on the first image to obtain at least one first object;

[0134] The second detection module 62 is used to perform target detection on the second image to obtain at least one second object; the first image is acquired based on the first camera, the second image is acquired based on the second camera, the field of view of the second camera is larger than the field of view of the first camera, and there is an overlap between the field of view of the second camera and the field of view of the first camera;

[0135] Extraction module 63 is used to extract the first image features and the first position features of each first object in the first image, and the second image features and the second position features of each second object in the second image;

[0136] The determination module 64 is used to determine the target object in the overlapping field of view of the first object and the second object based on the first image features and the first position features, as well as the second image features and the second position features.

[0137] The tracking module 65 is used to fuse the first image features and the second image features corresponding to the target object to obtain a third image feature, and to track the target object based on the third image feature.

[0138] The robot target tracking device in this embodiment performs target detection on a first image using a first detection module 61 to obtain at least one first object, and performs target detection on a second image using a second detection module 62 to obtain at least one second object. The first image is captured by a first camera, and the second image is captured by a second camera. The field of view of the second camera is larger than that of the first camera, and their fields of view overlap. That is, target detection is performed separately to obtain the objects to be matched from the two images captured by the two cameras. The first camera has a smaller field of view, while the second camera has a larger field of view, enabling the acquisition of the overall motion process of the objects to be matched. The extraction module 63 extracts the first image features and first position features of each first object in the first image, and the first image features of each second object in the second image. The second image feature and the second position feature in the image are used to obtain the image features and position features of the object to be matched; the determination module 64 determines the target object in the overlapping field of view of the first object and the second object based on the first image feature and the first position feature, as well as the second image feature and the second position feature, that is, it determines the target object that exists in the object to be matched, which is in both the first object and the second object; the tracking module 65 fuses the first image feature and the second image feature corresponding to the target object to obtain the third image feature, and tracks the target object based on the third image feature to obtain the fused image feature of the target object, which expands the tracking field of view of the target object, solves the problem in related technologies that robots have difficulty tracking the overall motion process of the tracked target, and improves the tracking effect.

[0139] In some embodiments, the determining module includes a first acquisition submodule, a second acquisition submodule, and a first determining submodule. The first acquisition submodule is used to acquire the image feature distance between each first image feature and each second image feature. The second acquisition submodule is used to acquire the position feature distance between each first position feature and each second position feature. The first determining submodule is used to determine the target object in the overlapping field of view of the first object and the second object based on the image feature distance and the position feature distance.

[0140] The robot target tracking device in this embodiment acquires image feature distances between each first image feature and each second image feature through a first acquisition submodule, and obtains a target object determination reference value from the image similarity between the first object and the second object; acquires position feature distances between each first position feature and each second position feature through a second acquisition submodule, and obtains a target object determination reference value from the position similarity between the first object and the second object; and determines the target object in the overlapping field of view between the first object and the second object based on each image feature distance and each position feature distance through a first determination submodule. By determining the target object in the first object and the second object using two reference values, the matching of tracking targets in different tracking fields of view is realized, and the tracking field of view of the target object is expanded.

[0141] In some embodiments, the first determining submodule includes a first constructing unit, a second constructing unit, a combining unit, and a determining unit. The first constructing unit is used to construct an image distance matrix based on the distances of each image feature; the second constructing unit is used to construct a position distance matrix based on the distances of each position feature; the combining unit is used to combine the image distance matrix and the position distance matrix to obtain a distance information matrix; and the determining unit is used to determine the target object in the overlapping field of view of the first object and the second object by using the value of the distance information matrix as a target object determination parameter.

[0142] The robot target tracking device in this embodiment constructs an image distance matrix based on the distances of various image features using a first construction unit, and uses the image distance matrix to describe the image feature distances between each first object and each second object. A second construction unit constructs a position distance matrix based on the distances of various position features, and uses the position distance matrix to describe the position feature distances between each first object and each second object. A combination unit combines the image distance matrix and the position distance matrix to obtain a distance information matrix, which is used to describe the comprehensive feature distances between each first object and each second object. A determination unit uses the values ​​of the distance information matrix as target object determination parameters to determine the target object within the overlapping field of view of the first and second objects. This achieves matching of multiple tracking targets in different tracking fields of view, expands the tracking field of view of the target object, and improves the tracking accuracy.

[0143] In some embodiments, the first position feature is the three-dimensional coordinates of the corresponding first object in the world coordinate system. The extraction module includes a second determination sub-module, which is used to determine the first position feature based on the transformation relationship between the robot coordinate system and the world coordinate system at the current moment, the transformation relationship between the first camera coordinate system and the robot coordinate system, and the homogeneous three-dimensional coordinates of the corresponding first object in the first camera coordinate system.

[0144] In this embodiment, the robot target tracking device performs coordinate system transformation between the world coordinate system, the robot coordinate system, and the first camera coordinate system through the second determination submodule. This transforms the homogeneous three-dimensional coordinates of the first object in the first camera coordinate system into homogeneous three-dimensional coordinates of the first object in the world coordinate system, thus determining a unified coordinate system for subsequent calculation of the position feature distance between the first object and the second object.

[0145] In some embodiments, the second position feature is the three-dimensional coordinates of the corresponding second object in the world coordinate system. The extraction module includes a third determination submodule, which is used to determine the second position feature based on the transformation relationship between the second camera and the world coordinate system and the homogeneous three-dimensional coordinates of the corresponding second object in the second camera coordinate system.

[0146] In this embodiment, the robot target tracking device performs coordinate system transformation between the world coordinate system and the second camera coordinate system through the third determination submodule. This transforms the homogeneous three-dimensional coordinates of the second object in the second camera coordinate system into homogeneous three-dimensional coordinates of the second object in the world coordinate system, thus determining a unified coordinate system for subsequent calculation of the position feature distance between the first object and the second object.

[0147] In some embodiments, the tracking module includes a concatenation submodule, which is used to concatenate the first image feature and the second image feature corresponding to the target object to obtain a third image feature.

[0148] The robot target tracking device in this embodiment connects the first image features and the second image features corresponding to the target object through a serial submodule to obtain a third image feature. This enhances the accuracy of target detection and recognition in the two fields of view of the first and second cameras, expands the tracking field of view of the target object, and realizes the tracking of the overall motion process of the target object.

[0149] In some embodiments, this application also provides a robot, which includes a first camera for acquiring a first image, and the robot target tracking device in the above embodiments.

[0150] In this embodiment, the robot acquires a first image using a first camera and obtains a second image. Based on the target detection results of the first and second images, it obtains a first object and a second object. Based on the image features and positional features of the first and second objects, it determines the target object within the overlapping field of view of the first and second images. It then fuses the image features of the first and second objects corresponding to the target object and tracks the target object based on the fused image features. This expands the tracking field of view of the target object, solves the problem in related technologies where robots have difficulty tracking the overall motion process of the tracked target, and improves the accuracy of tracking.

[0151] Furthermore, in conjunction with the robot target tracking methods provided in the above embodiments, this embodiment can also provide a storage medium for implementation. This storage medium stores a program; when executed by a processor, the program implements any of the robot target tracking methods described in the above embodiments.

[0152] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated in this embodiment.

[0153] It should be understood that the specific embodiments described herein are merely illustrative of the application and not intended to limit it. All other embodiments derived by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0154] Obviously, the accompanying drawings are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar situations based on these drawings without any creative effort. Furthermore, it is understood that although the work done in this development process may be complex and lengthy, for those skilled in the art, certain design, manufacturing, or production modifications made based on the technical content disclosed in this application are merely conventional technical means and should not be considered as insufficient disclosure of this application.

[0155] The term "embodiment" in this application refers to a specific feature, structure, or characteristic described in connection with an embodiment that may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily imply the same embodiment, nor does it imply that it is mutually exclusive with or independent of other embodiments. It will be clearly or implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0156] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of patent protection. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the appended claims.

Claims

1. A robot target tracking method, characterized by, The method comprises: target detection on a first image to obtain at least one first object; and target detection on a second image to obtain at least one second object; the first image is captured based on a first camera arranged on a robot body, the second image is captured based on a second camera arranged in an environment surrounding the robot or other equipment, a field of view range of the second camera is larger than that of the first camera and there is an overlap between the field of view ranges of the first and second cameras; extraction of first image features and first position features of each of the first objects in the first image and second image features and second position features of each of the second objects in the second image; determination of target objects in the first and second objects in the overlap field of view range based on the first image features and first position features and the second image features and second position features; fusion of the first image features and the second image features corresponding to the target objects to obtain third image features, and tracking of the target objects based on the third image features.

2. The method of claim 1, wherein, The determination of the target objects in the first and second objects in the overlap field of view range based on the first image features and first position features and the second image features and second position features comprises: acquisition of image feature distances between each of the first image features and each of the second image features; and acquisition of position feature distances between each of the first position features and each of the second position features; determination of the target objects in the first and second objects in the overlap field of view range based on each of the image feature distances and each of the position feature distances.

3. The method of claim 2, wherein, The determination of the target objects in the first and second objects in the overlap field of view range based on each of the image feature distances and each of the position feature distances comprises: construction of an image distance matrix based on each of the image feature distances; and construction of a position distance matrix based on each of the position feature distances; combination of the image distance matrix and the position distance matrix to obtain a distance information matrix; determination of the target objects in the first and second objects in the overlap field of view range based on values of the distance information matrix as target object determination parameters.

4. The method of claim 1, wherein, The first position features are three-dimensional coordinates of the corresponding first objects in a world coordinate system; the first position features are determined based on a conversion relationship between a robot coordinate system at a current time and the world coordinate system, a conversion relationship between a first camera coordinate system and the robot coordinate system, and homogeneous three-dimensional coordinates of the corresponding first objects in the first camera coordinate system.

5. The method of claim 1, wherein, The second position features are three-dimensional coordinates of the corresponding second objects in the world coordinate system; the second position features are determined based on a conversion relationship between the second camera and the world coordinate system and homogeneous three-dimensional coordinates of the corresponding second objects in a second camera coordinate system.

6. The method of claim 1, wherein, The fusion of the first image features and the second image features corresponding to the target objects to obtain third image features, and the tracking of the target objects based on the third image features comprises: The first image feature and the second image feature corresponding to the target object are concatenated to obtain the third image feature.

7. The method of claim 6, wherein, The concatenating the first image feature and the second image feature corresponding to the target object to obtain the third image feature comprises: F = n ([F i ; F j ]) wherein F is the third image feature; n() is a normalization function, F i is the first image feature corresponding to the i-th first object, F j is the second image feature corresponding to the j-th second object.

8. A robot target tracking apparatus characterized by comprising: The robot target tracking device comprises: A first detection module configured to perform target detection on a first image to obtain at least one first object; A second detection module configured to perform target detection on a second image to obtain at least one second object; the first image is captured based on a first camera arranged on a robot body, and the second image is captured based on a second camera arranged in an environment surrounding the robot or other equipment; a field of view range of the second camera is larger than that of the first camera and overlaps with the field of view range of the first camera; An extraction module configured to extract a first image feature and a first position feature of each of the first objects in the first image, and a second image feature and a second position feature of each of the second objects in the second image; A determination module configured to determine a target object in the first objects and the second objects based on the first image feature and the first position feature, and the second image feature and the second position feature; A tracking module configured to fuse the first image feature and the second image feature corresponding to the target object to obtain a third image feature, and track the target object based on the third image feature.

9. A robot, characterized in that The robot comprises: A first camera configured to capture a first image, and a robot target tracking device as claimed in claim 8.

10. A readable storage medium, having stored thereon a program, characterized in that, The program, when executed by a processor, implements the steps of the robot target tracking method as claimed in any one of claims 1 to 7. The program, when executed by a processor, implements the steps of the robot target tracking method as claimed in any one of claims 1 to 7.

Images