Target tracking method and device, storage medium and electronic device

By combining monocular vision and depth vision images, the three-dimensional feature information of the target is determined and the observation parameters are updated, and the problems of difficulty in solving three-dimensional coordinates and large errors in monocular vision tracking are solved, improving the accuracy of target tracking.

CN119027456APending Publication Date: 2024-11-26HUANENG JILIN POWER GENERATION JIUTAI ELECTRIC FACTORY +2
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411258679.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-09
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

During the target tracking process, monocular vision-based methods are prone to blind PnP problems when obtaining the three-dimensional coordinates of the target, which leads to difficulty in solving and large errors, especially when the target distance is far away, the distance information error is significant.

Method used

By acquiring monocular visual images and depth visual images, we determine the three-dimensional feature information of the target object, including distance information and depth information, use this information to calculate the observation parameters, and update the parameters when the observation parameters do not meet the preset difference conditions to perform state estimation. for accurate tracking.

Benefits of technology

The accuracy of target tracking is improved, combined with the image stability and anti-interference ability of the monocular camera and the resolution and depth measurement accuracy of the depth camera, the error is reduced and the accurate tracking of the target object is achieved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027456A_ABST
    Figure CN119027456A_ABST
Patent Text Reader

Abstract

The invention discloses a target tracking method and device, a storage medium and an electronic device, and the method comprises the steps: obtaining a monocular vision image and a depth vision image which are obtained through the image collection of a target object; based on the monocular vision image and the depth vision image, three-dimensional feature information of the target object is determined, and the three-dimensional feature information comprises distance information and target depth information of the target object; determining observation parameters of the target depth information and the distance information; and when the observation parameter does not meet a preset difference condition, updating the observation parameter, performing state estimation on the target object based on the updated observation parameter, and tracking the target object according to a state estimation result. The problem that the target tracking error is large in the prior art is solved, and the target tracking accuracy is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a target tracking method and device, a storage medium, and an electronic device. Background Art

[0002] With the development of visual applications, target tracking has become a basic and important task. There are many common methods for target tracking, such as tracking based on monocular vision, tracking based on depth vision, and tracking based on multi-sensor information fusion. These methods have their own advantages and disadvantages and are suitable for different application scenarios.

[0003] In the conventional target tracking process, tracking based on monocular vision is widely used. In the monocular vision tracking, in order to obtain the three-dimensional coordinates of the target, PnP is usually used.

[0004] (perspective-n-point) expansion, but this method will degenerate into a blind PnP problem when facing N 3D spatial points and their corresponding 2D projection positions are unknown, resulting in difficulty in solving and large errors. Especially when the target is far away, the distance information error obtained by P3P (a commonly used PnP method) is particularly significant.

[0005] Therefore, how to improve the accuracy of target tracking has become a problem that needs to be solved urgently. Summary of the invention

[0006] The embodiment of the present application provides a target tracking method to at least solve the problem of large target tracking error in the related art.

[0007] According to an embodiment of the embodiments of the present application, a target tracking method is provided, including: acquiring a monocular visual image and a depth visual image obtained by performing image acquisition on a target object; determining three-dimensional feature information of the target object based on the monocular visual image and the depth visual image, the three-dimensional feature information including distance information of the target object and target depth information; determining observation parameters of the target depth information and the distance information; when the observation parameters do not meet a preset difference condition, updating the observation parameters, and performing state estimation on the target object based on the updated observation parameters, and tracking the target object according to the state estimation result.

[0008] In an exemplary embodiment, determining the three-dimensional feature information of the target object based on the monocular vision image and the depth vision image includes: performing semantic segmentation on the monocular vision image to extract the target contour in the monocular vision image; determining feature pixel points from the target contour, and determining the monocular center coordinates of the target object in the monocular vision image based on the feature pixel points; performing coordinate transformation based on the monocular center coordinates to determine the depth center coordinates of the target object in the depth vision image; and determining the three-dimensional feature information of the target object based on the depth center coordinates.

[0009] In an exemplary embodiment, the coordinate transformation is performed based on the monocular center coordinates to determine the depth center coordinates of the target object in the depth vision image, including: determining the translation matrix and the rotation matrix between the monocular vision image and the depth vision image; based on the product of the translation matrix, the rotation matrix and the monocular center coordinates, determining the depth center coordinates of the target object in the depth vision image.

[0010] In an exemplary embodiment, determining the three-dimensional feature information of the target object based on the depth center coordinates includes: matching the depth center coordinates with the depth information of the depth image to obtain target depth information; determining the three-dimensional position of the target object based on the depth center coordinates, the target depth information, and a preset camera matrix; and calculating the distance information of the target object based on the three-dimensional position of the target object.

[0011] In an exemplary embodiment, the determination of observation parameters of the target depth information and the distance information includes: performing vector conversion on the target depth information and the distance information to obtain vector representations of the target depth information and the distance information; inputting the vector representation into a preset observation expression, and the observation expression determines the matrix representation of the target depth information and the distance information based on a set state transfer matrix, a measurement matrix, and noise parameters; and obtaining an observation covariance matrix based on the matrix representation.

[0012] In an exemplary embodiment, the observation parameters include an observation covariance matrix; when the observation parameters do not meet a preset difference condition, the observation parameters are updated, and a state estimation is performed on the target object based on the updated observation parameters, and the target object is tracked according to the state estimation result, including: when a difference between the observation covariance matrix and a preset target covariance matrix is ​​greater than a set difference threshold, determining that the observation parameters do not meet the preset difference condition; updating the observation covariance matrix to obtain an updated observation covariance matrix; calculating a state estimation of the target object according to the updated observation covariance matrix, and tracking the target object based on the state estimation.

[0013] In an exemplary embodiment, updating the observation covariance matrix to obtain an updated observation covariance matrix includes: determining a mapping matrix from state space to observation space, an error covariance matrix, and a forgetting factor; and using the mapping matrix, the error covariance matrix, the forgetting factor, and the mapping matrix to update the observation covariance matrix to obtain an updated observation covariance matrix.

[0014] According to another embodiment of the embodiments of the present application, a target tracking device is also provided, including: an acquisition module, used to acquire a monocular visual image and a depth visual image obtained by image acquisition for a target object; an analysis module, used to determine three-dimensional feature information of the target object based on the monocular visual image and the depth visual image, wherein the three-dimensional feature information includes distance information of the target object and target depth information; a determination module, used to determine observation parameters of the target depth information and the distance information; a tracking module, used to update the observation parameters when the observation parameters do not meet a preset difference condition, and perform state estimation on the target object based on the updated observation parameters, and track the target object according to the state estimation result.

[0015] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the above method when running.

[0016] According to another aspect of the embodiments of the present application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the method through the computer program.

[0017] According to another aspect of the embodiments of the present application, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0018] In the embodiment of the present application, by acquiring a monocular visual image and a depth visual image, the three-dimensional feature information of the target object can be determined based on the monocular visual image and the depth visual image. The three-dimensional feature information includes the distance information of the target object and the target depth information, and then the observation parameters of the target depth information and the distance information can be determined; when the observation parameters do not meet the preset difference conditions, the observation parameters are updated, and the state of the target object is estimated based on the updated observation parameters, and the target object is tracked according to the state estimation results. In the target tracking process, the three-dimensional feature information is determined based on the monocular visual image and the depth visual image, so that the advantages of the monocular camera in image stability, image transmission capability and anti-interference capability are utilized, and the advantages of the depth camera in resolution and depth measurement accuracy are utilized. On this basis, the observation parameters are determined based on the distance information and the target depth information, and when the observation parameters do not meet the preset error conditions, the observation parameters are updated, so as to expand the state estimation again, and realize accurate tracking of the target object. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0020] Figure 1 It is a hardware structure block diagram of a computer terminal of a target tracking method according to an embodiment of the present application;

[0021] Figure 2 is a flow chart of a target tracking method according to an embodiment of the present application;

[0022] Figure 3 is another flow chart of a target tracking method according to an embodiment of the present application;

[0023] Figure 4 is a schematic flow chart of measurement value distribution of a target tracking method according to an embodiment of the present application;

[0024] Figure 5 It is a structural block diagram of a target tracking device according to an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices; "plurality" means two or more.

[0027] The method embodiments provided in the embodiments of the present application can be executed in a computer terminal or similar computing device or cloud platform or independent physical server or software platform, wherein the above software platform is run through one or more servers. Taking running on a computer terminal as an example, Figure 1 is a hardware structure block diagram of a target tracking method according to an embodiment of the present application. Figure 1 As shown, the computer terminal may include one or more ( Figure 1 Only one is shown in the figure) processor 102 and memory 104 for storing data. In an exemplary embodiment, the computer terminal may also include a transmission device 106 for communication functions and an input and output device 108, wherein the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA. A person skilled in the art can understand that Figure 1 The structure shown is only for illustration and does not limit the structure of the above-mentioned computer terminal. Figure 1 More or fewer components as shown, or with Figure 1 Equivalent functions or comparisons shown Figure 1 A different configuration with more features is shown.

[0028] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as computer programs corresponding to the methods in the embodiments of the present application. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, that is, the above-mentioned methods are implemented. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0029] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of a computer terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0030] In this embodiment, a target tracking method is provided, which is applied to the above-mentioned computer terminal. Figure 2 : is a flow chart of a target tracking method according to an embodiment of the present application, the process includes the following steps:

[0031] Step S202, obtaining a monocular visual image and a depth visual image obtained by performing image acquisition on the target object;

[0032] It should be noted that the monocular vision image is obtained by performing target detection and acquisition on the target object with a monocular camera; the depth vision image is obtained by performing target detection and acquisition on the target object with a depth camera.

[0033] Step S204, determining three-dimensional feature information of the target object based on the monocular visual image and the depth visual image, wherein the three-dimensional feature information includes distance information and target depth information of the target object;

[0034] It is understandable that the computer terminal can use the monocular vision image and the depth vision image to perform binocular vision processing. The computer terminal can match and analyze the monocular vision image and the depth vision image to determine the three-dimensional feature information of the target object, including its distance information and target depth information. In this way, the three-dimensional feature information of the target object can be determined, providing important information support for subsequent target tracking, identification and positioning.

[0035] Step S206, inputting the target depth information and the distance information into a preset state observation expression, and using the state observation expression to perform state estimation on the target object based on the target depth information and the distance information to obtain observation parameters of the target object;

[0036] Specifically, the observation parameter is a parameter used to describe the error between the measured value and the standard value, and the observation parameter may include a covariance matrix of the target object.

[0037] Step S208, when the observation parameter does not meet the preset difference condition, the observation parameter is updated, and the state of the target object is estimated based on the updated observation parameter, and the target object is tracked according to the state estimation result.

[0038] The difference condition is a condition set for determining the error of an observation parameter. When the error between the observation parameter and the set target observation parameter is large, it is determined that the difference condition is satisfied.

[0039] In the embodiment of the present application, by acquiring a monocular visual image and a depth visual image, the three-dimensional feature information of the target object can be determined based on the monocular visual image and the depth visual image. The three-dimensional feature information includes the distance information of the target object and the target depth information, and then the observation parameters of the target depth information and the distance information can be determined; when the observation parameters do not meet the preset difference conditions, the observation parameters are updated, and the state of the target object is estimated based on the updated observation parameters, and the target object is tracked according to the state estimation results. In the target tracking process, the three-dimensional feature information is determined based on the monocular visual image and the depth visual image, so that the advantages of the monocular camera in image stability, image transmission capability and anti-interference capability are utilized, and the advantages of the depth camera in resolution and depth measurement accuracy are utilized. On this basis, the observation parameters are determined based on the distance information and the target depth information, and when the observation parameters do not meet the preset error conditions, the observation parameters are updated, so as to expand the state estimation again, and realize accurate tracking of the target object.

[0040] In an exemplary embodiment, determining the three-dimensional feature information of the target object based on the monocular vision image and the depth vision image includes: performing semantic segmentation on the monocular vision image to extract the target contour in the monocular vision image; determining feature pixel points from the target contour, and determining the monocular center coordinates of the target object in the monocular vision image based on the feature pixel points; performing coordinate transformation based on the monocular center coordinates to determine the depth center coordinates of the target object in the depth vision image; and determining the three-dimensional feature information of the target object based on the depth center coordinates.

[0041] It should be noted that for target recognition based on monocular vision: Yolov5 (you only look once, target detection algorithm) is used to realize target detection in monocular vision images; in the target detection process, the DeepLabV3 (deep convolutional neural network model for semantic segmentation) + improved segmentation head can be added to the head part of Yolov5 to realize semantic segmentation while detecting targets in Yolov5.

[0042] Among them, the feature pixel points are extracted from the target contour. The number of feature pixel points is 4, and the quadrilateral formed by the four feature pixel points is the maximum inscribed rectangle of the target contour. The geometric center of the four feature points is determined as the target center of the target object in the monocular vision image, and the coordinates of the target center are the monocular center coordinates.

[0043] In some embodiments, point O W , that is, the target center is located in the monocular vision pixel coordinate system. The calculation method of the monocular center coordinate is as follows. The monocular center coordinate includes the horizontal coordinate Vertical axis They are:

[0044]

[0045] Among them, u i and v i Refers to the coordinates of feature pixels, i=1, 2, 3, 4.

[0046] In the above embodiment, by processing the monocular vision image and the depth vision image separately, it is possible to determine the information of the target object corresponding to the monocular vision image, and to determine the information of the target object corresponding to the depth vision image, and then the two types of information can be fused subsequently, thereby improving the accuracy of target tracking.

[0047] In an exemplary embodiment, the coordinate transformation is performed based on the monocular center coordinates to determine the depth center coordinates of the target object in the depth vision image, including: determining the translation matrix and the rotation matrix between the monocular vision image and the depth vision image; based on the product of the translation matrix, the rotation matrix and the monocular center coordinates, determining the depth center coordinates of the target object in the depth vision image.

[0048] It should be noted that there is a translation matrix t and a rotation matrix R between the monocular camera and the depth camera system. When determining the translation matrix t and the rotation matrix R, the pixel coordinates of multiple groups of corresponding points in the depth camera and the monocular camera can be determined, and the translation matrix t and the rotation matrix R between the camera coordinate systems of the two cameras can be obtained by singular value decomposition.

[0049] In some embodiments, the formula for calculating the depth center coordinates is as shown in the following formula:

[0050]

[0051] in, are the depth center coordinates of the four feature pixels, P i are the monocular center coordinates of the four feature pixels, is the depth center coordinate of the target center in the depth camera pixel coordinate system, is the monocular center coordinate, and They are the inverse matrices of the intrinsic parameter matrices of the two cameras respectively.

[0052] In the above embodiment, the coordinate conversion between different images can be carried out through the translation matrix and the rotation matrix, so as to accurately determine the depth center coordinates and improve the accuracy of target tracking.

[0053] In an exemplary embodiment, determining the three-dimensional feature information of the target object based on the depth center coordinates includes: matching the depth center coordinates with the depth information of the depth image to obtain target depth information; determining the three-dimensional position of the target object based on the depth center coordinates, the target depth information, and a preset camera matrix; and calculating the distance information of the target object based on the three-dimensional position of the target object.

[0054] Specifically, after determining the depth center coordinates, the computer terminal can directly obtain the target depth information of the target object. When calculating the distance information, the projection matrix can be used to map the two-dimensional imaging coordinates in the image coordinate system to the three-dimensional position in the camera coordinate system, and the corresponding relationship between the imaging coordinates and the three-dimensional position can be obtained:

[0055] s[uv 1]=K[a|b][XYZ 1] T

[0056]

[0057] Among them, s is the scaling factor, K is the camera intrinsic parameter matrix, a and b are the camera extrinsic parameter matrices, [XYZ] is the three-dimensional position coordinate of the target object in the camera coordinate system, [uv] is the pixel coordinate of the target object in the depth camera coordinate system, 1 is the constant term in the homogeneous coordinate system, and the target depth information Z can directly obtain the depth value through the pixel coordinates of the corresponding point in the depth image.

[0058] The above formula can be simplified to:

[0059]

[0060] The three-dimensional position of the target object has been obtained through three-dimensional coordinate solution. When the camera intrinsic parameter matrix and distortion parameters are known, the monocular camera can solve the pose information of the target object in the world coordinate system, that is, the distance information of the target object in the camera coordinate system can be obtained.

[0061] In an exemplary embodiment, the determination of observation parameters of the target depth information and the distance information includes: performing vector conversion on the target depth information and the distance information to obtain vector representations of the target depth information and the distance information; inputting the vector representation into a preset observation expression, and the observation expression determines the matrix representation of the target depth information and the distance information based on a set state transfer matrix, a measurement matrix, and noise parameters; and obtaining an observation covariance matrix based on the matrix representation.

[0062] Specifically, the computer terminal can define the system state vector as:

[0063]

[0064] The state vector is a two-dimensional column vector x, which contains the projection component h of the monocular camera-target relative position vector on the camera optical axis and the projection component h of the monocular camera-target relative velocity vector on the camera optical axis.

[0065] It should be noted that the following discrete time observation equation, i.e., observation expression, is established for the distance information and depth information of the target object:

[0066]

[0067] H = [1 0]

[0068] In the discrete-time observation equation, That is, it is a matrix representation of the target depth information and the distance information, H is the measurement matrix; i=1 represents a depth camera, and i=0 represents a monocular camera.

[0069] In the above embodiment, by converting the target depth information and the distance information and combining them with a set observation expression, the observation covariance matrix of the target depth information and the distance information can be accurately determined.

[0070] In an exemplary embodiment, the observation parameters of the target object include an observation covariance matrix; when the observation parameters do not meet a preset difference condition, the observation parameters are updated, and the target object is tracked based on the updated observation parameters, including: when the difference between the observation covariance matrix and a preset target covariance matrix is ​​greater than a set difference threshold, determining that the observation parameters do not meet the preset difference condition; updating the observation covariance matrix to obtain an updated observation covariance matrix; calculating a state estimate of the target object according to the updated observation covariance matrix, and tracking the target object based on the state estimate.

[0071] It can be understood that when the difference between the observed covariance matrix and the set target covariance matrix is ​​greater than the set difference threshold, it means that the difference between the measured data and the target data is large. Therefore, the observed covariance matrix can be adjusted to reduce the difference between the measured data and the target data, so as to optimize asynchronous fusion and obtain the optimal state estimation of the target object.

[0072] In an exemplary embodiment, updating the observation covariance matrix to obtain an updated observation covariance matrix includes: determining a mapping matrix from state space to observation space, an error covariance matrix, and a forgetting factor; and using the mapping matrix, the error covariance matrix, the forgetting factor, and the mapping matrix to update the observation covariance matrix to obtain an updated observation covariance matrix.

[0073] It should be noted that the distance information calculated based on the three-dimensional position of the target object will face the problem of large errors and noise when the target is large. Therefore, the observation covariance matrix can be corrected. The specific correction formula is as follows:

[0074]

[0075] Among them, R k refers to the updated observation covariance matrix, ε k =y k -Hx k , ε kIt has no specific meaning and can be considered to be calculated from the target depth information and distance information. This formula introduces a forgetting factor 0<α<1 to adaptively adjust R k . H refers to the mapping matrix, P k|k-1 Refers to the error covariance matrix. The larger the α is, the more emphasis is placed on prior estimates and the longer time is needed to adapt to changes in the observed quantity. Since the system emphasizes the real-time tracking, the present application can set the forgetting factor α=0.3.

[0076] In some embodiments, the state estimate is calculated as follows:

[0077]

[0078] Among them, H k Represents the measurement matrix, matrix S k The singular value decomposition is used to solve the matrix, which has no practical significance. For S k The transposed matrix of y k is the measurement value determined by the distance information and the target depth information, and the Kalman filter’s prior estimate X k|k-1 and the error covariance matrix P k|k-1 , and the error covariance matrix P k|k . R k Refers to the observation covariance matrix. By updating the observation covariance matrix, the state estimation of the target object is obtained.

[0079] In the above embodiment, the state estimation can be made more accurate by continuously correcting and optimizing the observation covariance matrix.

[0080] It is understandable that in order to better understand the process of the above-mentioned target tracking method, the above-mentioned target tracking method is described below in combination with an optional embodiment, but it is not used to limit the technical solution of the embodiment of the present application.

[0081] Common methods for target tracking include tracking based on monocular vision, tracking based on depth vision, and tracking based on multi-sensor information fusion. Since the tracking method based on monocular vision lacks depth information, the tracking method based on monocular vision can only realize the target position estimation in the two-dimensional image plane, and cannot consider the state estimation problem of the target in the three-dimensional Cartesian space. In order to obtain the three-dimensional coordinates of the target position, the PnP (perspective-n-point) method is a commonly used method, but the PnP method has the problem that when n 3D space points and their positions are unknown, PnP degenerates into blind PnP. The use of tracking methods based on depth vision sensors can solve the above problems. However, although the depth camera can directly obtain depth information, the computational complexity of processing 3D point cloud data is very high, which makes it difficult to track the target in real time.

[0082] Monocular vision information is used to detect and segment the target, and a depth camera is used to obtain the target depth. Since the P3P solution has a large error in obtaining distance information when the target is far away, the Kalman filter is used to asynchronously fuse the distance information obtained by the monocular camera P3P solution and the depth information obtained by the depth camera. At the same time, dynamic weights are introduced to dynamically update the observation covariance matrix according to the measurement error. This not only utilizes the advantages of the monocular camera in image stability, image transmission capability and anti-interference ability, but also utilizes the advantages of the depth camera in resolution and depth measurement accuracy.

[0083] Firstly, monocular vision image information is used to realize target detection and complete target semantic segmentation. Then, monocular vision and depth vision are jointly calibrated, and the depth information of the identified target is obtained by the depth camera. Finally, based on the real-time and accuracy requirements of target tracking, the method of the present invention is designed to solve the problems that the error of monocular vision p3p solution is large when the target is far away and the recognition frame rate of the depth camera is slow. The distance information obtained by monocular vision and the depth information extracted by depth vision are asynchronously fused based on Kalman filtering to achieve accurate estimation of the target.

[0084] This application is mainly divided into the following three modules: (1) Target recognition based on monocular vision: The Yolo V5 (you on one look once) algorithm is used to achieve real-time target detection in color images. (2) Semantic segmentation based on monocular vision: The improved segmentation head based on DeepLabV3+ is added to the head part of Yolo V5 to achieve semantic segmentation while detecting targets. (3) Information fusion of monocular vision and deep vision.

[0085] Specifically, Figure 3 As shown, the flowchart of determining three-dimensional feature information in the optional embodiment of the present application is as follows: In the present application, it is composed of a tracking target, a monocular camera, and a depth camera. With each of them as the center, the tracking target establishes its own coordinate system in the Cartesian coordinate system, with the geometric center of the target object as the coordinate origin, and establishes a right-handed Cartesian coordinate system. T -X T Y T Z T The depth camera and the monocular industrial camera use the optical center of their respective camera lenses as the coordinate origin to establish the camera coordinate system, where the depth camera coordinate system is O G -X G Y G Z G , the coordinate system of the monocular industrial camera is O D -X D Y D Z DWhen shooting, the two cameras image the object on their respective imaging planes. The resulting coordinate system in the imaging plane is called the pixel coordinate system, which is pixel coordinate system u D -v D and u G -v G ,According to the pinhole imaging model, there is a one-to-one correspondence ,between the pixel coordinates on the camera imaging plane and the ,camera coordinates.

[0086] After semantic segmentation, the target contour has been extracted. Assume that the feature pixel point P on the contour i (u i ,v i )(i=1,2,3,4), and the four points connected to form a quadrilateral are the maximum inscribed rectangle of the contour. The geometric center of the four feature points, that is, the center of the maximum inscribed rectangle of the target contour, is taken as the origin of the world coordinate system O W

[0087] The target semantic segmentation is realized through the monocular camera, the target position is identified, and the target depth information at the target position is directly obtained by transforming the coordinates to the camera pixel coordinate system corresponding to the depth vision image. Therefore, the projection matrix can be used to map the two-dimensional imaging coordinates in the image coordinate system to the three-dimensional position in the camera coordinate system, and the corresponding relationship between the imaging coordinates and the three-dimensional position can be obtained.

[0088] After determining the three-dimensional feature information, when the camera's intrinsic parameter matrix and distortion parameters are known, the monocular camera can obtain the pose information of the monocular camera in the world coordinate system through P3P solution, that is, the distance information of the target object in the camera coordinate system can be obtained. This application can asynchronously fuse the distance information obtained by monocular vision with the depth information extracted by depth vision based on Kalman filtering to achieve accurate estimation of the target's motion state:

[0089] There exists a linear discrete system whose state space expression is:

[0090] X k+1 =A(k+1,k)X k +w k

[0091] Among them, X k Indicates that at time t = kT, T represents the sampling interval, matrix A is the system state transfer matrix, A(k+1,k) represents the state transfer matrix from time k to time k+1, w k represents Gaussian white noise with zero mean, and its covariance is the matrix Q k .

[0092] Assume that the system has M sensors with different sampling frequencies and initial sampling times, refer to Figure 4 As shown, define Ti is the sampling period of sensor i. In each time interval [(k-1)T, kT], a given sensor performs a total of N k At the same time, it is possible that a sensor did not measure or made one or more measurements during this time interval. Therefore, in the time interval [(k-1)T, kT], the following expression is established:

[0093]

[0094] in, represents the number of measurements provided by a sensor, and the measurements provided by all sensors are randomly distributed within this time interval. So the i-th measurement value can be expressed as:

[0095]

[0096] in, represents the measurement matrix of the sensor providing the i-th measurement value, v k represents Gaussian white noise with zero mean, and its covariance is the matrix R k .

[0097]

[0098] Therefore, the problem of asynchronous fusion can be simplified as: solving the conditional density function of the system state, that is, finding the optimal state estimate and the error covariance matrix of the optimal state estimate:

[0099] X k|k =E[X k |Y k ]

[0100] P k|k =cov[X k |Y k ]

[0101] for to k The state transfer matrix at time is defined as follows:

[0102]

[0103] Matrix A is the system state transfer matrix, A(k+1,k) represents the state transfer matrix from time k to time k+1. for to k The state transfer matrix at time Q k is the covariance matrix. Represents the measurement matrix of the sensor that provides the i-th measurement value. The matrix S is solved using singular value decomposition and has no practical significance.

[0104] Given the Kalman filter prior estimate X k|k-1 and the error covariance matrix P k|k-1 , and the optimal state estimate X under the centralized asynchronous fusion model k and the error covariance matrix P k|k :

[0105] X k|k-1 =A(k,k-1)X k-1

[0106] P k|k-1 =A(k,k-1)P k-1 A T (k,k-1)+Q(k,k-1)

[0107]

[0108] In summary, the optional embodiment of the present application can obtain a monocular visual image and a depth visual image, so that the three-dimensional feature information of the target object can be determined based on the monocular visual image and the depth visual image. The three-dimensional feature information includes the distance information of the target object and the target depth information, and then the observation parameters of the target depth information and the distance information can be determined; when the observation parameter does not meet the preset difference condition, the observation parameter is updated, and the state of the target object is estimated based on the updated observation parameter, and the target object is tracked according to the state estimation result. In the target tracking process, the three-dimensional feature information is determined based on the monocular visual image and the depth visual image, so that the advantages of the monocular camera in image stability, image transmission capability and anti-interference capability are utilized, and the advantages of the depth camera in resolution and depth measurement accuracy are utilized. On this basis, the observation parameter is determined based on the distance information and the target depth information, and when the observation parameter does not meet the preset error condition, the observation parameter will be updated, so as to expand the state estimation again, and realize accurate tracking of the target object.

[0109] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0110] The present application also provides a structural block diagram of a target tracking device. Figure 5 is a structural block diagram of a target tracking device according to an embodiment of the present application; Figure 5 As shown, including:

[0111] An acquisition module 52 is used to acquire a monocular visual image and a depth visual image obtained by performing image acquisition on a target object;

[0112] An analysis module 54, used to determine observation parameters of the target depth information and the distance information;

[0113] A determination module 56, used to determine observation parameters of the target depth information and the distance information;

[0114] The tracking module 58 is used to update the observation parameters when the observation parameters do not meet the preset difference conditions, and perform state estimation on the target object based on the updated observation parameters, and track the target object according to the state estimation results.

[0115] Through the above device, by acquiring a monocular visual image and a depth visual image, the three-dimensional feature information of the target object can be determined based on the monocular visual image and the depth visual image. The three-dimensional feature information includes the distance information of the target object and the target depth information, and then the observation parameters of the target depth information and the distance information can be determined; when the observation parameters do not meet the preset difference conditions, the observation parameters are updated, and the state of the target object is estimated based on the updated observation parameters, and the target object is tracked according to the state estimation results. In the target tracking process, the three-dimensional feature information is determined based on the monocular visual image and the depth visual image, so that the advantages of the monocular camera in image stability, image transmission capability and anti-interference capability are utilized, and the advantages of the depth camera in resolution and depth measurement accuracy are utilized. On this basis, the observation parameters are determined based on the distance information and the target depth information, and when the observation parameters do not meet the preset error conditions, the observation parameters are updated, so as to expand the state estimation again, and realize accurate tracking of the target object.

[0116] In an exemplary embodiment, the analysis module 54 is also used to perform semantic segmentation on the monocular vision image and extract the target contour in the monocular vision image; determine feature pixel points from the target contour, and determine the monocular center coordinates of the target object in the monocular vision image based on the feature pixel points; perform coordinate transformation based on the monocular center coordinates to determine the depth center coordinates of the target object in the depth vision image; and determine the three-dimensional feature information of the target object based on the depth center coordinates.

[0117] In an exemplary embodiment, the analysis module 54 is also used to determine the translation matrix and rotation matrix between the monocular vision image and the depth vision image; based on the product of the translation matrix, the rotation matrix and the monocular center coordinates, determine the depth center coordinates of the target object in the depth vision image.

[0118] In an exemplary embodiment, the analysis module 54 is also used to match the depth center coordinates with the depth information of the depth image to obtain target depth information; determine the three-dimensional position of the target object based on the depth center coordinates, the target depth information, and a preset camera matrix; and calculate the distance information of the target object based on the three-dimensional position of the target object.

[0119] In an exemplary embodiment, the determination module 56 is also used to perform vector conversion on the target depth information and the distance information to obtain vector representations of the target depth information and the distance information; input the vector representation into a preset observation expression, and the observation expression determines the matrix representation of the target depth information and the distance information based on a set state transfer matrix, a measurement matrix, and noise parameters; and obtain an observation covariance matrix based on the matrix representation.

[0120] In an exemplary embodiment, the tracking module 54 is also used to determine that the observation parameters do not meet a preset difference condition when the difference between the observation covariance matrix and the preset target covariance matrix is ​​greater than a set difference threshold; update the observation covariance matrix to obtain an updated observation covariance matrix; calculate a state estimate of the target object according to the updated observation covariance matrix, and track the target object based on the state estimate.

[0121] In an exemplary embodiment, the tracking module 54 is also used to determine a mapping matrix from state space to observation space, an error covariance matrix, and a forgetting factor; and use the mapping matrix, the error covariance matrix, the forgetting factor, and the mapping matrix to update the observation covariance matrix to obtain an updated observation covariance matrix.

[0122] An embodiment of the present application further provides a storage medium, which includes a stored program, wherein the program executes any of the above methods when it is run.

[0123] Optionally, in this embodiment, the storage medium may be configured to store program codes for executing the following steps:

[0124] S1, obtaining a monocular visual image and a depth visual image obtained by image acquisition of a target object;

[0125] S2, determining three-dimensional feature information of the target object based on the monocular visual image and the depth visual image, wherein the three-dimensional feature information includes distance information and target depth information of the target object;

[0126] S3, determining observation parameters of the target depth information and the distance information;

[0127] S4, when the observation parameter does not meet the preset difference condition, the observation parameter is updated, and the state of the target object is estimated based on the updated observation parameter, and the target object is tracked according to the state estimation result.

[0128] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0129] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0130] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0131] S1, obtaining a monocular visual image and a depth visual image obtained by image acquisition of a target object;

[0132] S2, determining three-dimensional feature information of the target object based on the monocular visual image and the depth visual image, wherein the three-dimensional feature information includes distance information and target depth information of the target object;

[0133] S3, determining observation parameters of the target depth information and the distance information;

[0134] S4, when the observation parameter does not meet the preset difference condition, the observation parameter is updated, and the state of the target object is estimated based on the updated observation parameter, and the target object is tracked according to the state estimation result.

[0135] Optionally, in this embodiment, the above-mentioned storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store program codes.

[0136] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any one of the above method embodiments are implemented.

[0137] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0138] An embodiment of the present application also provides a computer program, which includes computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps in any one of the above method embodiments.

[0139] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0140] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0141] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. A target tracking method, characterized in that: include: Acquire a monocular vision image and a depth vision image obtained by image acquisition of a target object; Determine three-dimensional feature information of the target object based on the monocular visual image and the depth visual image, wherein the three-dimensional feature information includes distance information and target depth information of the target object; Determining observation parameters of the target depth information and the distance information; When the observation parameter does not satisfy a preset difference condition, the observation parameter is updated, and a state estimation is performed on the target object based on the updated observation parameter, and the target object is tracked according to the state estimation result.

2. The method according to claim 1, characterized in that The determining of the three-dimensional feature information of the target object based on the monocular visual image and the depth visual image includes: Performing semantic segmentation on the monocular visual image to extract the target contour in the monocular visual image; Determine characteristic pixel points from the target contour, and determine the monocular center coordinates of the target object in the monocular vision image according to the characteristic pixel points; Performing coordinate transformation based on the monocular center coordinates to determine the depth center coordinates of the target object in the depth vision image; Based on the depth center coordinates, three-dimensional feature information of the target object is determined.

3. The method according to claim 2, characterized in that The performing coordinate conversion based on the monocular center coordinate to determine the depth center coordinate of the target object in the depth vision image includes: Determining a translation matrix and a rotation matrix between the monocular vision image and the depth vision image; Based on the product of the translation matrix, the rotation matrix and the monocular center coordinates, the depth center coordinates of the target object in the depth vision image are determined.

4. The method according to claim 2, characterized in that: The determining the three-dimensional feature information of the target object based on the depth center coordinates includes: Matching the depth center coordinates with the depth information of the depth image to obtain target depth information; Determine the three-dimensional position of the target object based on the depth center coordinates, the target depth information, and a preset camera matrix; The distance information of the target object is calculated based on the three-dimensional position of the target object.

5. The method according to claim 1, characterized in that The determining of the observation parameters of the target depth information and the distance information includes: Performing vector conversion on the target depth information and the distance information to obtain vector representations of the target depth information and the distance information; Inputting the vector representation into a preset observation expression, the observation expression determines the matrix representation of the target depth information and the distance information based on a set state transfer matrix, a measurement matrix and a noise parameter; Based on the matrix representation, an observation covariance matrix is ​​obtained.

6. The method according to claim 1, characterized in that The observation parameters include an observation covariance matrix; When the observation parameter does not meet the preset difference condition, the observation parameter is updated, and a state estimation is performed on the target object based on the updated observation parameter, and the target object is tracked according to the state estimation result, including: When the difference between the observed covariance matrix and the preset target covariance matrix is ​​greater than a set difference threshold, determining that the observed parameter does not meet the preset difference condition; Updating the observation covariance matrix to obtain an updated observation covariance matrix; A state estimate of the target object is calculated according to the updated observation covariance matrix, and the target object is tracked based on the state estimate.

7. The method according to claim 6, characterized in that The updating of the observation covariance matrix to obtain an updated observation covariance matrix includes: Determine the mapping matrix from state space to observation space, the error covariance matrix, and the forgetting factor; The mapping matrix, the error covariance matrix, the forgetting factor and the mapping matrix are used to update the observation covariance matrix to obtain an updated observation covariance matrix.

8. A target tracking device, characterized in that: include: An acquisition module is used to acquire a monocular visual image and a depth visual image obtained by performing image acquisition on a target object; An analysis module, configured to determine three-dimensional feature information of the target object based on the monocular visual image and the depth visual image, wherein the three-dimensional feature information includes distance information and target depth information of the target object; A determination module, used to determine observation parameters of the target depth information and the distance information; The tracking module is used to update the observation parameters when the observation parameters do not meet the preset difference conditions, and to perform state estimation on the target object based on the updated observation parameters, and to track the target object according to the state estimation results.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein the program executes the method described in any one of claims 1 to 7 when executed.

10. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 7 through the computer program.

Citation Information

Cited By

  • Visual tracking tool quality detection method and controller

    CN120628545A