Target tracking method, system, device, equipment and medium based on RGB-D image

Abstract

The application provides a target tracking method, system, device, equipment and medium based on an RGB-D image, wherein a current frame RGB-D image in video data collected by a depth camera is acquired, pedestrian detection is performed on the current frame RGB image, a current frame RGB target detection box containing a tracking object is obtained, a current frame actual barycenter point of the tracking object is obtained according to the current frame RGB target detection box and a current frame Depth image, and a target tracking strategy is formulated according to the current frame actual barycenter point and a pre-stored current frame predicted barycenter point. That is, the application not only considers the RGB image, but also takes the Depth image in the RGB-D image as a reference, and the depth information is not affected by light, shadow and the like, so that the calculated actual barycenter point of the tracking object is more accurate, and then a more accurate target tracking strategy can be formulated according to the more accurate current frame actual barycenter point and the pre-stored current frame predicted barycenter point, so as to improve the stability and accuracy of target tracking.

Description

Technical Field

[0001] This invention relates to the field of multimedia technology, and specifically to target tracking methods, systems, devices, equipment, and media based on RGB-D images. Background Technology

[0002] Pedestrian detection technology is widely used in information statistics and related emergency measures in public places, such as pedestrian flow control, shopping mall layout reference, and public security. This technology not only has broad application prospects and great potential in intelligent monitoring systems, but it is also an attractive and challenging problem in computer vision. Visual analysis of pedestrian movement is an emerging and cutting-edge research field, involving multiple areas such as intelligent assisted driving, motion capture, intelligent monitoring, human behavior recognition and analysis, and environmental control and monitoring.

[0003] Currently, the commonly used method is to use RGB images for detection and then tracking. In the motion prediction process of tracking, it is necessary to calculate the pedestrian's center of gravity and associate the motion based on the pedestrian's center of gravity. The pedestrian's center of gravity is represented by the center point calculated from the detected pedestrian bounding box, and the pedestrian's center of gravity can be used for motion prediction in pedestrian tracking.

[0004] However, when using the above-mentioned common methods to track the human body, since pedestrians are non-rigid bodies and have different shapes, when the pedestrian's limbs extend and move, even if the pedestrian's position does not change, the pedestrian frame will change. At this time, the calculated center of gravity will also change, but in fact the pedestrian has not moved. Therefore, the above-mentioned method for calculating the center of gravity will have problems of instability and inaccuracy in practical applications.

[0005] Therefore, existing technologies need to be improved. Summary of the Invention

[0006] The main objective of this invention is to propose a target tracking method, system, device, equipment, and medium based on RGB-D images, so as to at least solve the technical problem of inaccuracy in existing target tracking methods.

[0007] A first aspect of the present invention provides a target tracking method based on RGB-D images, comprising:

[0008] Acquire the current frame RGB-D image from the video data captured by the depth camera; wherein, the current frame RGB-D image includes the current frame RGB image and the current frame Depth image; perform pedestrian detection on the current frame RGB image to obtain a current frame RGB target detection box containing the tracked object; obtain the current frame actual centroid point of the tracked object based on the current frame RGB target detection box and the current frame Depth image; predict the motion trajectory of the tracked object based on the current frame actual centroid point to obtain the predicted centroid point of the next frame and store it; formulate a target tracking strategy based on the current frame actual centroid point and the pre-stored current frame predicted centroid point.

[0009] Furthermore, a second aspect of the present invention provides a target tracking device, the target tracking device comprising an acquisition module, a detection module, a centroid detection module, a prediction module, and a strategy formulation module; the acquisition module is used to acquire the current frame RGB-D image from video data captured by a depth camera; wherein, the current frame RGB-D image includes a current frame RGB image and a current frame Depth image; the detection module is used to perform pedestrian detection on the current frame RGB image to obtain a current frame RGB target detection box containing the tracked object; the centroid detection module is used to obtain the current frame actual centroid point of the tracked object based on the current frame RGB target detection box and the current frame Depth image; the prediction module is used to predict the motion trajectory of the tracked object based on the current frame actual centroid point, obtain the predicted centroid point for the next frame, and store it; the strategy formulation module is used to formulate a target tracking strategy based on the current frame actual centroid point and the pre-stored current frame predicted centroid point.

[0010] In addition, a third aspect of the present invention provides a target tracking system, including a depth camera, a memory, a processor, and a computer program stored in the memory and executable on the processor; the depth camera is used to acquire video data, the video data consisting of consecutive RGB-D images; the processor executes the computer program to implement the steps in the target tracking method based on RGB-D images provided in the first aspect.

[0011] In addition, a fourth aspect of the present invention provides an electronic device, including a memory, a processor, and a bus;

[0012] The bus is used to enable communication between the memory and the processor;

[0013] The processor is used to execute computer programs stored in the memory;

[0014] When the processor executes the computer program, it implements the steps in the target tracking method based on RGB-D images provided in the first aspect.

[0015] In addition, a fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, characterized in that, when the computer program is executed by a processor, it implements the steps in the target tracking method based on RGB-D images provided in the first aspect.

[0016] This invention provides a target tracking method, system, apparatus, device, and medium based on RGB-D images. It acquires the current frame RGB-D image from video data captured by a depth camera, performs pedestrian detection on the current frame RGB image to obtain a current frame RGB target detection box containing the tracked object, obtains the current frame's actual centroid point of the tracked object based on the current frame RGB target detection box and the current frame depth image, predicts the motion trajectory of the tracked object based on the current frame's actual centroid point, obtains the predicted centroid point for the next frame and stores it, and formulates a target tracking strategy based on the current frame's actual centroid point and the pre-stored current frame predicted centroid point. In other words, this application, in its implementation, considers not only the RGB image but also the depth image within the RGB-D image as a reference. Since using depth information is unaffected by lighting, shadows, etc., it ensures that the calculated actual centroid point of the tracked object is more accurate. Therefore, based on this more accurate current frame actual centroid point and the pre-stored current frame predicted centroid point, a more accurate target tracking strategy can be formulated, thereby improving the stability and accuracy of target tracking. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the target tracking system provided in an embodiment of the present invention;

[0019] Figure 2 This is a flowchart illustrating the target tracking method based on RGB-D images provided in an embodiment of the present invention;

[0020] Figure 3 This is a flowchart illustrating the target tracking method based on RGB-D images provided in an embodiment of the present invention;

[0021] Figure 4 This is a schematic diagram of projecting a foreground image onto a ground plane and obtaining scattered points projected onto the ground plane in an embodiment of the present invention;

[0022] Figure 5This is a flowchart illustrating the target tracking method based on RGB-D images provided in an embodiment of the present invention;

[0023] Figure 6 This is a schematic diagram illustrating the relationship between the image coordinate system and the pixel coordinate system in an embodiment of the present invention;

[0024] Figure 7 This is a schematic diagram illustrating the relationship between the camera coordinate system and the image coordinate system in an embodiment of the present invention;

[0025] Figure 8 This is a schematic diagram of the module connections inside an electronic device provided in an embodiment of the present invention.

[0026] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0027] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0028] It should be noted that related terms such as "first" and "second" can be used to describe various components, but these terms do not limit the component. These terms are only used to distinguish one component from another. For example, without departing from the scope of the invention, the first component can be referred to as the second component, and the second component can similarly be referred to as the first component. The term "and / or" refers to any one or more combinations of related and descriptive terms.

[0029] Please see Figure 1 , Figure 1 This invention illustrates a target tracking system provided by an embodiment of the present invention. The target tracking system includes a depth camera 11, a memory 12, a processor 13, and a computer program stored in the memory 12 and executable on the processor 13.

[0030] Specifically, the depth camera 11 includes structured light depth cameras, binocular depth cameras, time-of-flight depth cameras, and depth cameras with RGB cameras (i.e., RGB-D cameras), which are used to acquire video data, which consists of continuous RGB-D images.

[0031] Specifically, memory 12 represents a device with storage function, used to store computer programs. Processor 13 represents a device with processing and execution functions, used to execute the computer program on memory 12 to implement a target tracking method based on RGB-D images. Therefore, when the above target tracking system is applied to public places, it can track various targets in crowds.

[0032] For the target tracking method based on RGB-D images mentioned above, please refer to [link / reference]. Figure 2 The target tracking method specifically includes the following steps:

[0033] Step S10: Obtain the RGB-D image of the current frame from the video data captured by the depth camera;

[0034] Specifically, when a depth camera is installed in a shopping mall area for pedestrian flow statistics, it can acquire the current frame RGB-D image from the video data collected by the depth camera. The current frame RGB-D image includes the current frame RGB image and the current frame Depth image. Therefore, when the first RGB image is obtained, the corresponding first Depth image is also obtained simultaneously. Generally, the current frame image represents the image corresponding to the current moment.

[0035] An RGB-D image can include both an RGB image and a Depth image. Essentially, an RGB-D image consists of two images: a standard RGB three-channel color image and a Depth image. The RGB image describes the appearance, color, and texture information of an object, while the Depth image describes the spatial depth information of the object.

[0036] The following explains the definitions and relationships between RGB images and Depth images:

[0037] The RGB color mode of RGB images is an industry color standard. It obtains a variety of colors by changing the three color channels of red (R), green (G), and blue (B) and superimposing them. RGB represents the colors of the three channels of red, green, and blue. This standard includes almost all colors that human vision can perceive and is one of the most widely used color systems.

[0038] In 3D computer graphics, a depth map is an image or image channel that contains information about the distances to the surfaces of scene objects from the viewpoint. A depth map is similar to a grayscale image, except that each pixel value represents the actual distance from the sensor to the object. Typically, RGB images and depth images are registered, resulting in a one-to-one correspondence between pixels.

[0039] Step S20: Perform pedestrian detection on the current frame RGB image to obtain the current frame RGB target detection box containing the tracked object;

[0040] Specifically, a pedestrian detector can be used to detect pedestrians in the current frame of the RGB image. This pedestrian detector is a neural network model that uses computer vision technology to determine whether pedestrians exist in the image. That is, after the pedestrian detector performs target detection on the current frame of the RGB image, an RGB target detection box will appear in the current frame of the RGB image, thus obtaining the current frame RGB target detection box. This RGB target detection box uses computer vision technology to determine whether pedestrians exist in the image; in other words, the pedestrian detector is used to detect the tracked object in the RGB image, and the current frame RGB target detection box is used to locate the tracked object.

[0041] Step S30: Obtain the actual centroid of the tracked object in the current frame based on the RGB target detection box and the current frame depth image;

[0042] Specifically, if the target detection box of the current frame appears in the current frame RGB image, the actual centroid of the tracked object in the current frame can be calculated based on the target detection box and the current frame depth image. It can be understood that, since the current frame RGB image is referenced on the basis of the original current frame depth image, the characteristics of the depth image can be used to avoid the influence of lighting, shadows, etc., thus ensuring that the calculated actual centroid of the tracked object in the current frame is more accurate.

[0043] Step S40: Based on the actual center of gravity of the current frame, predict the motion trajectory of the tracked object, obtain the predicted center of gravity of the next frame, and store it.

[0044] Specifically, after obtaining the actual centroid point of the current frame, the Kalman filter algorithm can be used to predict the motion trajectory of the tracked object, thereby obtaining the predicted centroid point of the next frame. It should be noted that the actual centroid point of the current frame indicates the actual centroid point of the tracked object at the current moment, while the predicted centroid point of the next frame indicates the predicted and estimated centroid point of the tracked object at the next moment. After predicting the predicted centroid point of the next frame, the obtained predicted centroid point is saved (it is used for the analysis and comparison process of the tracked object at the next moment).

[0045] Kalman filtering (KF) is an algorithm that uses the state equations of a linear system to make an optimal estimate of the system state using the system's input and output observation data.

[0046] Step S50: Formulate a target tracking strategy based on the actual centroid of the current frame and the pre-stored predicted centroid of the current frame.

[0047] Specifically, the actual centroid of the current frame reflects the actual position of the centroid of the tracked object in the current frame's RGB image at the current moment. The predicted centroid of the current frame is obtained from the previous frame's RGB-D image (the frame before the current frame). Specifically, the motion trajectory of the tracked object is predicted based on the actual centroid of the previous RGB-D image, resulting in the predicted centroid of the current frame, which is then stored. This reflects the predicted position of the tracked object's centroid in the previous RGB-D image at the next moment. Then, these two positions are used to formulate an appropriate target tracking strategy. For example, if the difference between the actual and predicted centroids is large (significant positional difference), it is highly likely that the detected tracked object is not the same, and a strategy to adjust the tracking object should be formulated. Conversely, if the difference between the actual and predicted centroids is small or almost unchanged (minimal positional difference), it is highly likely that the detected tracked object is the same, and a strategy to continue tracking that object should be formulated.

[0048] By implementing the above embodiments, since the depth image is additionally referenced in the original RGB image, it can solve problems that pure RGB images cannot solve to a certain extent. For example, using depth information can be unaffected by lighting, shadows, etc., thus ensuring that the calculated actual center of gravity of the tracked object is more accurate. Furthermore, based on the more accurate actual center of gravity of the current frame and the pre-stored predicted center of gravity of the current frame, a more accurate target tracking strategy can be formulated, thereby improving the stability and accuracy of target tracking.

[0049] In this embodiment, the step of formulating a target tracking strategy based on the actual centroid of the current frame and the pre-stored predicted centroid of the current frame specifically includes:

[0050] Calculate the distance between the actual centroid point of the current frame and the predicted centroid point of the current frame, and determine whether the distance is within the preset distance range;

[0051] If the distance value is within a preset distance range, a tracking strategy is formulated to maintain the marker corresponding to the tracked object;

[0052] If the distance value is not within the preset distance range, a tracking strategy is formulated to change the marker corresponding to the tracked object.

[0053] Specifically, after obtaining the actual centroid of the current frame, based on the pre-stored predicted centroid of the current frame, the distance between the actual centroid and the predicted centroid can be calculated using these two centroids. This distance is then compared with a preset distance range. If the distance falls within the preset range, it indicates that the tracking object corresponding to the actual centroid and the tracking object corresponding to the predicted centroid are the same target. In this case, a tracking strategy is adopted to maintain the marker corresponding to the tracking object, i.e., the marker corresponding to the tracking object is not changed. If the distance does not fall within the preset range, it indicates that the tracking object corresponding to the actual centroid and the tracking object corresponding to the predicted centroid are different targets. In this case, a tracking strategy is adopted to change the marker corresponding to the tracking object, i.e., the marker is updated for the target corresponding to the actual centroid of the current frame.

[0054] In this embodiment, obtaining the actual centroid point of the tracked object in the current frame based on the current frame RGB target detection box and the current frame Depth image includes:

[0055] Map the RGB object detection box of the current frame to the Depth image of the current frame to obtain the Depth object detection box of the current frame;

[0056] Specifically, since the size of the current frame RGB image is the same as the size of the current frame Depth image, the position coordinates of the current frame RGB target detection box obtained on the current frame RGB image do not need to be changed, and can be directly mapped to the current frame Depth image, that is, the current frame Depth target detection box is obtained on the current frame Depth image.

[0057] Calculate the centroid of the tracked object within the current frame's Depth target detection bounding box to obtain the actual centroid point of the tracked object in the current frame.

[0058] Specifically, there are two schemes to calculate the centroid of the Depth object detection box and obtain the actual centroid point of the tracked object in the current frame. The first scheme is to convert the current frame Depth object detection box in the current frame Depth image into a point cloud, cluster the point clouds, and then calculate the centroid of each pedestrian. However, this method has a large computational load and is easily affected by pedestrian pose.

[0059] Therefore, a second solution is proposed: see here. Figure 3 and Figure 4 The steps for calculating the centroid of the tracked object within the current frame's Depth object detection bounding box and obtaining the actual centroid point of the tracked object in the current frame include the following:

[0060] Step S301: Obtain the ground plane and foreground images based on the current frame Depth image;

[0061] Specifically, the ground plane can be obtained by performing plane fitting on the current frame's depth image, such as using the RANSAC (Random Sample Consensus) algorithm or the least squares algorithm to perform plane fitting, thereby obtaining a well-fitted ground plane; and the foreground image can be obtained by performing preset foreground-background separation on the current frame's depth image.

[0062] Step S302: Obtain the scattered points of the foreground image projected onto the ground plane (see reference). Figure 4 );

[0063] Specifically, the benefits of projecting the foreground image onto the ground are: on the one hand, it can reduce the influence of pedestrian posture, and on the other hand, it can use the depth map to distinguish the center of gravity of pedestrians in front and behind, and calculate the center of gravity of multiple pedestrians at the same time.

[0064] Step S303: Perform cluster analysis on the scatter points and calculate the centroid coordinates of each cluster;

[0065] Step S304: Calculate the target centroid point of the current frame Depth target detection box based on the centroid coordinates to obtain the actual centroid point of the tracked object in the current frame RGB image.

[0066] The target centroid point of the Depth object detection box is obtained based on the centroid coordinates, and the target centroid point is used as the actual centroid point of the tracked object in the current frame of the RGB image.

[0067] Specifically, after performing cluster analysis on the scattered points projected onto the ground plane, the scattered points can be classified according to the depth information to distinguish between pedestrians in front and behind. Since each category of scattered points corresponds to a target, the centroid coordinates of each category can be calculated accordingly. The centroid point of the target detection box in the current frame can be calculated from the centroid coordinates, which is to obtain the actual centroid point of the tracked object in the current frame RGB image.

[0068] In this embodiment, before the step of obtaining the actual centroid point of the tracked object in the current frame based on the current frame RGB target detection box and the current frame Depth image, the method further includes:

[0069] Calculate the average pixel value of each target detection box in the current frame Depth image; delete target detection boxes in the current frame whose average pixel value is less than a preset pixel threshold.

[0070] Specifically, after obtaining the current frame depth image, the average pixel value of each target detection box in the current frame depth image is calculated. If the average pixel value of the target detection box is less than a preset pixel threshold, the target in the current frame depth target detection box is considered to be roughly a human shadow (a human shadow should not be tracked), and the target detection boxes with average pixel values ​​less than the preset pixel value are deleted. This reduces the interference of shadows caused by lighting in the RGB image. The average pixel value in the depth image refers to the average distance from the image acquisition device to each point in the scene, which directly reflects the geometry of the visible surface of the target in the image.

[0071] In this embodiment, the steps of obtaining the ground plane and foreground image based on the current frame Depth image specifically include:

[0072] Step M: Fit the ground plane to the current frame Depth image to obtain the fitted ground plane;

[0073] Step N: Perform morphological operations and background modeling on the current frame Depth image to obtain the foreground image.

[0074] Specifically, the methods for fitting the ground plane to the current frame depth image include using the RANSAC (Random Sample Consensus) algorithm and the least squares algorithm; among them, morphological operations are used to change the shape of the tracked target in the current frame depth image, and background modeling represents the establishment of a background model, which can be used to detect the tracked target in the current frame depth image.

[0075] It should be noted that there is no specific limitation on the order of steps M and N. Step M can be before step N, step N can be before step M, or steps N and M can be performed simultaneously.

[0076] Please see Figure 5 The steps of calculating the centroid point of the target detection box in the current frame based on the centroid coordinates, and obtaining the actual centroid point of the tracked object in the current frame RGB image, specifically include:

[0077] Step S501: Map the centroid coordinates to the current frame Depth image to obtain the depth value and image coordinates of each target centroid point on the current frame Depth image;

[0078] Specifically, after obtaining the centroid coordinates, the centroid coordinates can be mapped onto the current frame's depth image to obtain the centroid's image coordinates (x, y, z), where x and y represent the x-axis and y-axis position information, respectively, and z represents the depth value. The resulting image coordinates will have more depth values ​​than the centroid coordinates, thus more accurately reflecting the actual position of the tracked object.

[0079] Step S502: Establish a transformation matrix between depth values ​​and camera coordinate system based on the intrinsic parameter matrix of the depth camera;

[0080] Step S503: Transform the image coordinates to the world coordinate system according to the transformation matrix, obtain the world coordinates of the target centroid, and obtain the actual centroid of the tracked object in the current frame.

[0081] Specifically, after establishing the transformation matrix, the image coordinates are transformed to the world coordinate system using the transformation matrix, and then the world coordinates of the target centroid can be obtained. The actual centroid of the tracked object in the current frame can be obtained through these world coordinates.

[0082] The following section elaborates on the intrinsic parameter matrix, transformation matrix, image coordinate system, world coordinate system, and related coordinate transformation methods involved in steps S502-S503:

[0083] Specifically, camera intrinsic parameters are parameters related to the camera's own characteristics, such as the camera's focal length and pixel size; camera extrinsic parameters are parameters in the world coordinate system, such as the camera's position and rotation direction. The camera's extrinsic parameter matrix includes rotation and translation matrices, so the camera's extrinsic parameters are also called rotation and translation matrices. The rotation matrix is ​​used to describe the direction of the coordinate axes of the world coordinate system relative to the camera coordinate axes, and the translation matrix is ​​used to describe the position of the origin in the camera coordinate system.

[0084] For details, please refer to Figure 6 The origin of the image coordinate system xy is O1, which is the midpoint of the pixel coordinate system;

[0085] Assuming (u0, v0) represents the coordinates of the origin O1 in the uv coordinate system, and dx and dy represent the physical dimensions of each pixel on the horizontal axis (x) and vertical axis (y), respectively; then the relationship between the image coordinate system and the pixel coordinate system is as follows:

[0086]

[0087] Furthermore, assuming the unit in the physical coordinate system is millimeters, then the unit of dx is millimeters per pixel, and the unit of x / dx is pixels, which is the same as the unit of u. For convenience, the above two relationships can be written in matrix form:

[0088]

[0089] in, This is the intrinsic parameter matrix of the depth camera.

[0090] The inverse relation of the matrix formed by combining the image coordinate system and the pixel coordinate system can be expressed as:

[0091]

[0092] Please see Figure 7 It shows the relationship between the camera coordinate system and the image coordinate system; where O is the optical center of the depth camera, and Z... c Let be the optical axis of the depth camera, perpendicular to the image plane; OO1 is the focal length of the depth camera; and the four coordinate systems for camera calibration and their relationships are as follows:

[0093]

[0094]

[0095]

[0096] Therefore, based on the above relationships, it can be seen that a transformation matrix between depth values ​​and the camera coordinate system can indeed be established based on the intrinsic parameter matrix of the depth camera, and the image coordinates can be transformed to the world coordinate system based on the transformation matrix to obtain the world coordinates of the target centroid, which is to finally obtain the actual centroid of the tracked object in the current frame.

[0097] The present invention also provides a target tracking device, comprising: an acquisition module for acquiring a current frame RGB-D image from video data captured by a depth camera; wherein the current frame RGB-D image includes a current frame RGB image and a current frame Depth image; a detection module for performing pedestrian detection on the current frame RGB image to obtain a current frame RGB target detection box containing the tracked object; a centroid detection module for obtaining the current frame actual centroid point of the tracked object based on the current frame RGB target detection box and the current frame Depth image; a prediction module for predicting the motion trajectory of the tracked object based on the current frame actual centroid point to obtain and store the predicted centroid point for the next frame; and a strategy formulation module for formulating a target tracking strategy based on the current frame actual centroid point and the pre-stored current frame predicted centroid point.

[0098] Based on the aforementioned target tracking device, since a depth image is additionally referenced in the original RGB image, it can solve problems that pure RGB images cannot solve to a certain extent. For example, using depth information can avoid the influence of lighting and shadows. Secondly, by using depth information and employing methods such as background modeling and clustering to calculate the pedestrian's center of gravity, it can better represent the actual position of the pedestrian, reduce the influence of the pedestrian's posture, and thus ensure the stability and accuracy of the center of gravity, thereby improving the stability and accuracy of tracking.

[0099] Figure 8 An electronic device provided in an embodiment of the present invention is shown. This electronic device can be used to implement the target tracking method based on RGB-D images in any of the foregoing embodiments. The electronic device includes:

[0100] The system includes a memory 801, a processor 802, a bus 803, and a computer program stored on the memory 801 and executable on the processor 802. The memory 801 and the processor 802 are connected via the bus 803. When the processor 802 executes the computer program, it implements the target tracking method based on RGB-D images described in the foregoing embodiments. The number of processors can be one or more.

[0101] The memory 801 can be a high-speed random access memory (RAM) or a non-volatile memory, such as a disk storage device. The memory 801 is used to store executable program code, and the processor 802 is coupled to the memory 801.

[0102] Furthermore, embodiments of this application also provide a computer-readable storage medium, which may be disposed in the electronic device in the above embodiments, and the computer-readable storage medium may be a memory.

[0103] The computer-readable storage medium stores a computer program that, when executed by a processor, implements the target tracking method based on RGB-D images described in the foregoing embodiments. Furthermore, the computer-readable storage medium can also be various media capable of storing program code, such as a USB flash drive, external hard drive, read-only memory (ROM), RAM, magnetic disk, or optical disk.

[0104] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0105] The modules described as separate components may or may not be physically separate. Similarly, the components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0106] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0107] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned readable storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0108] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0109] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0110] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A target tracking method based on RGB-D images, characterized in that, include: Acquire the current frame RGB-D image from the video data captured by the depth camera; wherein, the current frame RGB-D image includes the current frame RGB image and the current frame Depth image; Perform pedestrian detection on the current frame RGB image to obtain the current frame RGB target detection box containing the tracked object; Map the current frame RGB target detection box to the current frame Depth image to obtain the current frame Depth target detection box; The ground plane and foreground image are obtained based on the current frame Depth image. Scattered points of the foreground image projected onto the ground plane are obtained. Cluster analysis is performed on the scattered points, and the centroid coordinates of each class are calculated. Calculate the target centroid point of the current frame Depth target detection box based on the centroid coordinates, and obtain the actual centroid point of the tracked object in the current frame RGB image of the current frame. Based on the actual center of gravity of the current frame, the motion trajectory of the tracked object is predicted, the predicted center of gravity of the next frame is obtained and stored; A target tracking strategy is formulated based on the actual centroid of the current frame and the pre-stored predicted centroid of the current frame.

2. The target tracking method based on RGB-D images as described in claim 1, characterized in that, Before the step of obtaining the actual centroid point of the tracked object in the current frame based on the current frame RGB target detection box and the current frame depth image, the method further includes: Calculate the average pixel value of each target detection box in the current frame Depth image; Delete the target detection boxes in the current frame whose average pixel value is less than a preset pixel threshold.

3. The target tracking method based on RGB-D images as described in claim 1, characterized in that, The step of calculating the target centroid point of the current frame Depth target detection box based on the centroid coordinates, and obtaining the actual centroid point of the tracked object in the current frame RGB image, includes: Map the centroid coordinates to the current frame depth image to obtain the depth value and image coordinates of each target centroid point on the current frame depth image; Establish a transformation matrix between depth values ​​and the camera coordinate system based on the intrinsic parameter matrix of the depth camera; The image coordinates are transformed to the world coordinate system according to the transformation matrix, the world coordinates of the target centroid are obtained, and the actual centroid of the tracked object in the current frame is obtained.

4. The target tracking method based on RGB-D images as described in claim 1, characterized in that, The step of formulating a target tracking strategy based on the actual centroid of the current frame and the pre-stored predicted centroid of the current frame specifically includes: Calculate the distance between the actual centroid point of the current frame and the predicted centroid point of the current frame, and determine whether the distance value is within a preset distance range; If the distance value is within a preset distance range, a target tracking strategy is formulated to maintain the marker corresponding to the tracked object; If the distance value is not within the preset distance range, a target tracking strategy is formulated to change the marker corresponding to the tracked object.

5. A target tracking device, characterized in that, include: The acquisition module is used to acquire the current frame RGB-D image from the video data captured by the depth camera; wherein, the current frame RGB-D image includes the current frame RGB image and the current frame Depth image; The detection module is used to perform pedestrian detection on the current frame RGB image to obtain the current frame RGB target detection box containing the tracked object; The centroid detection module is used to map the current frame RGB target detection box to the current frame Depth image to obtain the current frame Depth target detection box, obtain the ground plane and foreground image based on the current frame Depth image, obtain the scatter points projected from the foreground image onto the ground plane, perform cluster analysis on the scatter points, calculate the centroid coordinates of each class, calculate the target centroid point of the current frame Depth target detection box based on the centroid coordinates, and obtain the current frame actual centroid point of the tracked object in the current frame RGB image; The prediction module is used to predict the motion trajectory of the tracked object based on the actual center of gravity of the current frame, obtain the predicted center of gravity of the next frame, and store it. The strategy formulation module is used to formulate a target tracking strategy based on the actual centroid of the current frame and the pre-stored predicted centroid of the current frame.

6. A target tracking system, characterized in that, The target tracking system includes a depth camera, a memory, a processor, and a computer program stored in the memory and capable of running on the processor; The depth camera is used to acquire video data, which consists of continuous RGB-D images; When the processor executes the computer program, it implements the target tracking method as described in any one of claims 1 to 4.

7. An electronic device, characterized in that, Includes memory, processor, and bus; The bus is used to enable communication between the memory and the processor; The processor is used to execute computer programs stored in the memory; When the processor executes the computer program, it implements the steps of the target tracking method according to any one of claims 1 to 4.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the target tracking method according to any one of claims 1 to 4.

Images