Monocular vision 3D target tracking method, system, storage medium and device

Abstract

This invention discloses a monocular vision-based 3D target tracking method, system, storage medium, and device. The method includes: acquiring an initial image and a real-time acquired image of the target object; the initial image includes a projected contour and a simplified symbolic distance function (SDM) graph; segmenting the real-time acquired image using the SDM to obtain segmented regions; calculating the probability model of the region to which any pixel belongs within the segmented region; the segmented regions include foreground and background regions; calculating the composite energy of the real-time acquired image, which includes contour energy and texture energy; and updating the 3D pose based on the composite energy function; and extracting the final 3D pose output, wherein the output result includes the final 3D position and pose of the currently acquired image. This invention effectively solves the 3D tracking challenges of monocular vision in high-speed, occluded, and dynamic scenes, possessing advantages such as high real-time performance, strong robustness, and high integration, and is suitable for various industrial vision control scenarios.

Description

Technical Field

[0001] This invention relates to the field of visual processing technology, and more specifically, to a monocular vision three-dimensional target tracking method, system, storage medium, and device. Background Technology

[0002] In modern industrial applications such as industrial robots, automated manufacturing, intelligent logistics, and drone tracking, real-time and accurate acquisition of the three-dimensional position and attitude information of moving targets is one of the key technologies for achieving high-precision control, dynamic grasping, and autonomous navigation.

[0003] Traditional 3D localization methods typically rely on sensor systems such as multi-view stereo vision, depth cameras, LiDAR, or structured light. While these systems offer high accuracy, they have limitations in terms of hardware cost, system complexity, real-time performance, and environmental adaptability. Particularly in tracking high-speed moving targets, traditional vision systems, due to their low frame rate and high processing latency, struggle to provide real-time feedback on target position and orientation, thus limiting the robot's ability to operate in high-speed, dynamic environments.

[0004] With the development of visual sensors and computing hardware, monocular vision systems have gradually become a research hotspot in the field of industrial vision due to their advantages such as simple structure, low cost, and ease of deployment. Monocular vision systems can acquire two-dimensional image information of a target using only a single camera, and infer the target's position and pose in three-dimensional space through image processing and 3D reconstruction algorithms. However, monocular vision systems also face inherent technical challenges, especially in the absence of depth information; how to achieve stable, robust, and high-speed 3D pose estimation remains a problem that urgently needs to be solved. Summary of the Invention

[0005] The purpose of this invention is to provide a monocular vision three-dimensional target tracking method, system, storage medium, and device to solve the problem that existing visual tracking methods cannot simultaneously meet the requirements of high precision, high speed, and strong robustness.

[0006] The first aspect of this invention provides a monocular vision three-dimensional target tracking method, comprising the following steps:

[0007] Acquire an initial image and a real-time acquired image of the target object, wherein the initial image includes a projected contour and a simplified symbolic distance function graph;

[0008] The real-time acquired image is segmented using a simplified symbolic distance function to obtain segmented regions. A probability model of the region to which any pixel belongs within the segmented region is calculated. The segmented regions include foreground and background regions.

[0009] The composite energy of the real-time acquired image is calculated, which includes contour energy and texture energy, and the 3D pose is updated based on the composite energy function.

[0010] The final 3D pose is extracted and output, including the final 3D position and pose of the currently acquired image.

[0011] In this solution, obtaining the initial image of the target object specifically includes:

[0012] When the system starts, it acquires the original image and initial pose of the target object;

[0013] The original image is format-converted, and an image pyramid is constructed based on the format-converted original image;

[0014] The projected contour and the simplified symbolic distance function graph are obtained by combining the three-dimensional mesh model of the target object and the initial pose for contour rendering and symbolic distance function simplification.

[0015] In this solution, contour rendering and simplification of the symbolic distance function are performed, specifically including:

[0016] The three-dimensional mesh model is projected onto a virtual image plane according to the initial pose to obtain the projected contour.

[0017] Each pixel is judged based on the projection contour. If the pixel is within the projection contour, the symbolic distance function value is the first value; otherwise, it is the second value.

[0018] In this scheme, the real-time acquired image is segmented using a simplified signed distance function to obtain segmented regions. A probability model of the region to which any pixel within the segmented region belongs is calculated, specifically including:

[0019] The foreground and background regions of the real-time acquired image are obtained by segmenting the image using a simplified symbolic distance function.

[0020] Calculate statistical histograms of the color values ​​of all pixels in the foreground and background regions, respectively.

[0021] The probability model of the region to which any pixel belongs is calculated based on the statistical histogram, wherein the probability model includes the foreground probability of any pixel belonging to the foreground region and the background probability of any pixel belonging to the background region.

[0022] In this scheme, the composite energy of the real-time acquired image is calculated, which includes contour energy and texture energy. The 3D pose is then updated based on the composite energy function, specifically including:

[0023] The contour energy is obtained by measuring the consistency between the projected contour and the real image segmentation based on the probability model and the simplified symbolic distance function graph.

[0024] Within the projected contour region, the sum of squared texture differences between the corresponding regions of the current frame image and the previous frame image is calculated to obtain the texture energy.

[0025] The composite energy is obtained by proportionally summing the contour energy and the texture energy using a preset composite energy function;

[0026] The target matrix of the composite energy function with respect to the pose parameters is calculated using the Gauss-Newton method, and the update amount of the target matrix is ​​solved to update the three-dimensional pose.

[0027] A second aspect of the present invention also provides a monocular vision three-dimensional target tracking system, including a memory and a processor. The memory includes a monocular vision three-dimensional target tracking method program, which, when executed by the processor, performs the following steps:

[0028] Acquire an initial image and a real-time acquired image of the target object, wherein the initial image includes a projected contour and a simplified symbolic distance function graph;

[0029] The real-time acquired image is segmented using a simplified symbolic distance function to obtain segmented regions. A probability model of the region to which any pixel belongs within the segmented region is calculated. The segmented regions include foreground and background regions.

[0030] The composite energy of the real-time acquired image is calculated, which includes contour energy and texture energy, and the 3D pose is updated based on the composite energy function.

[0031] The final 3D pose is extracted and output, including the final 3D position and pose of the currently acquired image.

[0032] In this solution, obtaining the initial image of the target object specifically includes:

[0033] When the system starts, it acquires the original image and initial pose of the target object;

[0034] The original image is format-converted, and an image pyramid is constructed based on the format-converted original image;

[0035] The projected contour and the simplified symbolic distance function graph are obtained by combining the three-dimensional mesh model of the target object and the initial pose for contour rendering and symbolic distance function simplification.

[0036] In this solution, contour rendering and simplification of the symbolic distance function are performed, specifically including:

[0037] The three-dimensional mesh model is projected onto a virtual image plane according to the initial pose to obtain the projected contour.

[0038] Each pixel is judged based on the projection contour. If the pixel is within the projection contour, the symbolic distance function value is the first value; otherwise, it is the second value.

[0039] In this scheme, the real-time acquired image is segmented using a simplified signed distance function to obtain segmented regions. A probability model of the region to which any pixel within the segmented region belongs is calculated, specifically including:

[0040] The foreground and background regions of the real-time acquired image are obtained by segmenting the image using a simplified symbolic distance function.

[0041] Calculate statistical histograms of the color values ​​of all pixels in the foreground and background regions, respectively.

[0042] The probability model of the region to which any pixel belongs is calculated based on the statistical histogram, wherein the probability model includes the foreground probability of any pixel belonging to the foreground region and the background probability of any pixel belonging to the background region.

[0043] In this scheme, the composite energy of the real-time acquired image is calculated, which includes contour energy and texture energy. The 3D pose is then updated based on the composite energy function, specifically including:

[0044] The contour energy is obtained by measuring the consistency between the projected contour and the real image segmentation based on the probability model and the simplified symbolic distance function graph.

[0045] Within the projected contour region, the sum of squared texture differences between the corresponding regions of the current frame image and the previous frame image is calculated to obtain the texture energy.

[0046] The composite energy is obtained by proportionally summing the contour energy and the texture energy using a preset composite energy function;

[0047] The target matrix of the composite energy function with respect to the pose parameters is calculated using the Gauss-Newton method, and the update amount of the target matrix is ​​solved to update the three-dimensional pose.

[0048] A third aspect of the present invention provides a computer-readable storage medium comprising a machine program for a monocular vision three-dimensional target tracking method, wherein when executed by a processor, the monocular vision three-dimensional target tracking method program implements the steps of a monocular vision three-dimensional target tracking method as described in any of the preceding claims.

[0049] A fourth aspect of the present invention provides a monocular vision three-dimensional target tracking device, applied to the monocular vision three-dimensional target tracking method described in any of the preceding claims, wherein the device comprises:

[0050] The system includes a high-speed monocular RGB camera, an image processing unit, a control unit, and an active gimbal system.

[0051] The high-speed monocular RGB camera includes an industrial camera used to acquire two-dimensional image sequences of high-speed moving target objects;

[0052] The image processing unit is used to calculate the pose information of the high-speed moving target object from the two-dimensional image sequence using a preset algorithm;

[0053] The control unit is used to calculate the adjustment angle of the multi-degree-of-freedom gimbal based on the pose information and output control commands.

[0054] The active gimbal system includes a multi-degree-of-freedom gimbal controlled by a drive motor, used to support the industrial camera and adjust the shooting angle of the industrial camera based on the control commands.

[0055] This invention discloses a monocular vision 3D target tracking method, system, storage medium, and device, which effectively solves the 3D tracking challenges of monocular vision in high-speed, occluded, and dynamic scenes. It possesses advantages such as high real-time performance, strong robustness, and high integration, and is suitable for various industrial vision control scenarios. Specific beneficial effects are as follows:

[0056] 1. High-speed processing capability, with a processing speed of up to "400Hz", far exceeding the traditional PWP3D (25-30Hz).

[0057] 2. Strong robustness, capable of stable tracking even in partially occluded scenarios;

[0058] 3. Fast convergence, reducing the number of iterations from "6-8" to "3-4";

[0059] 4. High integration, realizing a vision-control closed-loop system, with industrial application capabilities;

[0060] 5. Highly adaptable, it can adapt to different texture targets by adjusting the texture energy weight. Attached Figure Description

[0061] Figure 1 A flowchart of a monocular vision three-dimensional target tracking method according to the present invention is shown;

[0062] Figure 2 A block diagram of a monocular vision three-dimensional target tracking system according to the present invention is shown;

[0063] Figure 3A block diagram of a monocular vision three-dimensional target tracking device according to the present invention is shown. Detailed Implementation

[0064] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0065] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0066] In the field of monocular vision 3D pose estimation, a variety of algorithms have been proposed. Among them, region-based 3D target tracking methods have attracted widespread attention due to their robustness to changes in texture and illumination. One of the most representative methods is the pixel-wise posterior probability-based 3D tracking algorithm (PWP3D), which has become one of the benchmark algorithms for real-time 3D pose estimation in monocular vision.

[0067] The core idea of ​​the PWP3D algorithm is to model the 3D pose estimation problem as a nonlinear optimization problem based on contour energy. The algorithm first generates the projected contour of the target object on the image plane by rendering its 3D model, and then embeds the contour into the image segmentation process using a level-set function. Subsequently, it constructs pixel-level statistical models of the foreground and background regions to calculate the probability of each pixel belonging to the foreground or background. Based on this, PWP3D defines a contour energy function that measures the consistency between the current projected contour and the real image contour. By minimizing this energy function, it iteratively updates the 3D pose (including position and rotation) of the target object using optimization algorithms such as Gauss-Newton, thereby achieving real-time pose estimation.

[0068] The PWP3D algorithm demonstrates good robustness on multiple public datasets, maintaining high tracking accuracy even in environments with significant lighting variations. Furthermore, because the algorithm relies solely on target contour information and not texture features, it reduces the requirements for image quality to some extent. However, the PWP3D algorithm also has significant limitations:

[0069] 1. High computational complexity: The PWP3D algorithm requires analytical calculation of the level set function, with a time complexity of O(n), and it is difficult to achieve fully parallel processing at the pixel level. Although some research has attempted to improve computational efficiency through GPU acceleration, its processing speed is still insufficient to meet the real-time requirements of high-speed vision systems (such as those above 500Hz).

[0070] 2. Sensitive to partial occlusion: Since the PWP3D algorithm relies solely on contour information for pose estimation, when the target object is partially occluded, its projected contour may break or deform, causing the energy function optimization process to get stuck in a local optimum or even fail to track.

[0071] 3. Limited convergence speed: In high dynamic scenes, the pose of the target object changes significantly between adjacent frames. Due to the limited optimization step size, the PWP3D algorithm may need to be iterated multiple times to converge, thus limiting its applicability in high frame rate image sequences.

[0072] Besides PWP3D, there are other 3D pose estimation methods based on feature points, template matching, or deep learning. For example, feature point-based methods extract features such as corners, edges, or ORBs from images and establish a correspondence between 2D image points and 3D model points, then solve for the pose using the PnP algorithm. These methods perform well in scenes with rich textures, but tend to fail in scenes with missing or repetitive textures. Deep learning methods, on the other hand, rely on large amounts of labeled data for training, often facing challenges in practical industrial applications such as difficulty in data acquisition and insufficient model generalization ability.

[0073] In summary, although various monocular vision 3D pose estimation methods have been proposed, existing methods still struggle to simultaneously meet the requirements of high accuracy, high speed, and strong robustness in complex industrial scenarios such as high speed, high dynamic range, and partial occlusion. Specifically, PWP3D and its derivatives suffer from the following main drawbacks in industrial high-speed vision applications:

[0074] 1. Insufficient Real-Time Performance: The PWP3D algorithm has high computational complexity. Although some research has improved its processing speed to 25–30Hz using GPU acceleration, this speed is still insufficient for real-time feedback and control in high-speed tracking scenarios for industrial robots (such as above 500Hz). The bottleneck mainly lies in the inability to fully parallelize the analytical calculation process of the level set function, resulting in low utilization of computing resources.

[0075] 2. Sensitive to partial occlusion: The PWP3D algorithm optimizes based solely on contour energy. When the target object is partially occluded, contour information is lost or distorted, causing the energy function to fail to guide the optimization process correctly, resulting in tracking drift or failure. This is very common in industrial environments, such as when a workpiece is obstructed by a fixture during robotic arm gripping, or when objects on a conveyor belt overlap.

[0076] 3. Strong dependence on initial pose: The PWP3D algorithm is highly sensitive to the initial pose. If the initial pose deviates significantly from the true pose, the optimization process is prone to getting trapped in local optima and converging to an incorrect pose. This problem is particularly prominent in high-dynamic scenes, where pose changes are significant between adjacent frames.

[0077] 4. Insufficient utilization of texture information: The PWP3D algorithm only uses pixels in the contour region during optimization, ignoring the texture information of the target object's surface. In scenes with rich textures, texture information can effectively supplement contour information, improving the stability and accuracy of pose estimation.

[0078] To address the shortcomings of the existing technologies, the present invention aims to provide a monocular vision-based 3D target tracking method and system. While maintaining the robustness of the PWP3D algorithm, through algorithm optimization and hardware co-design, the following objectives are achieved: Increased processing speed: By designing a simplified level set function suitable for parallel computing and fully utilizing the parallel computing capabilities of GPUs, the processing speed of 3D pose estimation is increased to over 400Hz, meeting the requirements of high-speed visual feedback; Enhanced robustness to partial occlusion: Based on contour energy, a texture energy term is introduced to construct a composite energy function, enabling the system to still perform stable tracking based on texture information even when the target contour is partially occluded; Improved convergence speed and stability: Through the design of the composite energy function and a multi-level image pyramid structure, the convergence process of pose estimation is optimized, enabling it to converge quickly and stably to the true pose in high-speed image sequences; System integration and real-time control: The vision algorithm is integrated with the robot control system to construct a complete active vision platform, achieving real-time tracking and localization of the 3D position and pose of high-speed moving targets.

[0079] Specifically, Figure 1 A flowchart of a monocular vision 3D target tracking method according to this application is shown.

[0080] like Figure 1 As shown, this application discloses a monocular vision three-dimensional target tracking method, including the following steps:

[0081] S102, acquire an initial image and a real-time acquired image of the target object, wherein the initial image includes a projected contour and a simplified symbolic distance function graph;

[0082] S104, the real-time acquired image is segmented using a simplified symbolic distance function to obtain a segmented region, and the probability model of the region to which any pixel belongs within the segmented region is calculated;

[0083] S106, calculate the composite energy of the real-time acquired image, the composite energy including contour energy and texture energy, and update the three-dimensional pose based on the composite energy function;

[0084] S108, extract the final 3D pose and output it. The output result includes the final 3D position and pose of the currently acquired image.

[0085] It should be noted that, in this embodiment, the present invention proposes two core improvements based on the traditional algorithm, namely, simplifying the level set function and the composite energy function. First, the initial image of the target object (high-speed moving target) and the real-time acquired image are obtained. Both the initial image and the real-time acquired image are captured by an industrial camera. The initial image is used to obtain the projection contour and the symbolic distance function as the initial state, while the real-time acquired image is used to update the pose in real time based on the initial state. The specific details will be described in detail in the subsequent specification.

[0086] According to an embodiment of the present invention, obtaining the initial image of the target object specifically includes:

[0087] When the system starts, it acquires the original image and initial pose of the target object;

[0088] The original image is format-converted, and an image pyramid is constructed based on the format-converted original image;

[0089] The projected contour and the simplified symbolic distance function graph are obtained by combining the three-dimensional mesh model of the target object and the initial pose for contour rendering and symbolic distance function simplification.

[0090] It should be noted that, in this embodiment, when the system starts, it acquires the original image of the target object and directly transmits the image data to the GPU global memory through high-speed interfaces such as the PCIe bus, avoiding the CPU copy bottleneck. The original image is then converted to a different format, for example, converting the Bayer array image to an RGB image. Different resolution versions of the original image (e.g., 1:1, 1:2, 1:4) are generated to form an image pyramid structure. The top layer has a low resolution for quick, rough estimation, while the bottom layer has a high resolution for fine optimization. In practical applications, an industrial camera can be used to acquire images at a frame rate of 425. The acquired image data is directly transmitted to the GPU global memory via a preset onboard bus for preprocessing operations such as de-mosaicing and image pyramid construction. Furthermore, the initial 3D pose of the target (6 degrees of freedom: X, Y, Z, rotation Rx, Ry, Rz) is provided by the user or an external system at system startup. This initial 3D pose is used to generate the rendering contour of the first frame, serving as the initial pose for the current frame.

[0091] According to an embodiment of the present invention, contour rendering and simplification of the symbolic distance function are specifically included:

[0092] The three-dimensional mesh model is projected onto a virtual image plane according to the initial pose to obtain the projected contour.

[0093] Each pixel is judged based on the projection contour. If the pixel is within the projection contour, the symbolic distance function value is the first value; otherwise, it is the second value.

[0094] It should be noted that, in this embodiment, the traditional PWP3D algorithm uses an analytical level set function, which has high computational complexity and is difficult to parallelize. Therefore, this invention proposes a simplified Signed Distance Function (SDF), defined as follows:

[0095]

[0096] in, and The adjustable parameters represent the SDF values ​​of the foreground and background regions, respectively. This simplified SDF no longer precisely calculates the distance from each pixel to the contour, but instead assigns fixed negative and positive values ​​to the entire foreground and background regions, respectively. In one embodiment, the first value is a negative value "-1" and the second value is a positive value "+1". Although this simplification introduces some quantization error, in high frame rate image sequences, the impact of this error on the optimization process is negligible because the target pose changes little between adjacent frames. Furthermore, since this SDF calculation process is fully parallelized, it can fully utilize the parallel computing power of the GPU, significantly improving processing speed.

[0097] Specifically, in practical applications, the OpenGL / DirectX graphics API can be used to project the 3D model onto a virtual image plane according to its pose, generating its accurate projection contour and depth map. Based on the rendered contour, each pixel in the image is judged. If the pixel is within the projection contour, the signed distance function value is the first value; otherwise, it is the second value.

[0098] According to an embodiment of the present invention, the real-time acquired image is segmented using a simplified symbolic distance function to obtain segmented regions, and a probability model of the region to which any pixel belongs within the segmented region is calculated, specifically including:

[0099] The foreground and background regions of the real-time acquired image are obtained by segmenting the image using a simplified symbolic distance function.

[0100] Calculate statistical histograms of the color values ​​of all pixels in the foreground and background regions, respectively.

[0101] The probability model of the region to which any pixel belongs is calculated based on the statistical histogram, wherein the probability model includes the foreground probability of any pixel belonging to the foreground region and the background probability of any pixel belonging to the background region.

[0102] It should be noted that, in this embodiment, based on the pixel information of the current image, a statistical histogram model of the foreground and background regions is constructed, and the probability of each pixel belonging to the foreground or background is calculated. Specifically, the real-time acquired image is clearly divided into a foreground region (target object) and a background region using a simplified signed distance function, and statistical histograms of all pixel color values ​​(such as RGB or grayscale) in the foreground and background regions are calculated respectively. Based on the histograms, the probability of any pixel value belonging to the foreground region and the probability of belonging to the background region are calculated.

[0103] According to an embodiment of the present invention, the composite energy of the real-time acquired image is calculated, the composite energy including contour energy and texture energy, and the three-dimensional pose is updated based on the composite energy function, specifically including:

[0104] The contour energy is obtained by measuring the consistency between the projected contour and the real image segmentation based on the probability model and the simplified symbolic distance function graph.

[0105] Within the projected contour region, the sum of squared texture differences between the corresponding regions of the current frame image and the previous frame image is calculated to obtain the texture energy.

[0106] The composite energy is obtained by proportionally summing the contour energy and the texture energy using a preset composite energy function;

[0107] The target matrix of the composite energy function with respect to the pose parameters is calculated using the Gauss-Newton method, and the update amount of the target matrix is ​​solved to update the three-dimensional pose.

[0108] It should be noted that, in this embodiment, in order to enhance the robustness of the algorithm to partial occlusion, the present invention introduces a texture energy term on the basis of traditional contour energy, and constructs the following composite energy function:

[0109] ;

[0110] in, For pixels, For position, Contour energy, defined in the same way as traditional PWP3D, is used to measure the consistency between the projected contour and the real contour. Texture energy is defined as the sum of squares of texture differences within the target region between adjacent frames, and is calculated as follows:

[0111] ;

[0112] in, For pixels, For position, and They represent the first Frame and the Texture information of the target region in the frame image. This is a Heaviside function used to confine texture differences within the target contour area; This is the weighting coefficient for texture energy, adjusted based on the texture richness of the target object: for objects with rich textures, it is recommended to use [weighting coefficient]. For objects with few textures, it is recommended to use... .

[0113] Furthermore, the composite energy function considers both contour consistency and texture consistency during the optimization process, ensuring that the system can still perform stable tracking based on texture information even when the target contour is partially occluded. Specifically, in the iterative optimization, the Gauss-Newton method is used to calculate the composite energy function for the pose parameters. The target matrix is ​​the Jacobian matrix, and the update amount of the target matrix is ​​solved. To update and obtain the three-dimensional pose This process proceeds from coarse to fine on the image pyramid, first converging rapidly at the low-resolution layer, and then using the result as the initial value for the high-resolution layer for fine optimization.

[0114] Furthermore, the three-dimensional pose is finally extracted and output. The output results include the final three-dimensional position and orientation of the currently acquired image. Specifically, the pose data (usually a 4x4 transformation matrix or a 6-dimensional vector) is transferred from the GPU memory to the control unit (CPU) for output. This output is the final result of the entire algorithm process and is directly transmitted to the active gimbal system or robot for real-time closed-loop control.

[0115] Figure 2 A block diagram of a monocular vision three-dimensional target tracking system according to the present invention is shown.

[0116] like Figure 2 As shown, this invention discloses a monocular vision 3D target tracking system, including a memory and a processor. The memory includes a monocular vision 3D target tracking method program, which, when executed by the processor, performs the following steps:

[0117] Acquire an initial image and a real-time acquired image of the target object, wherein the initial image includes a projected contour and a simplified symbolic distance function graph;

[0118] The real-time acquired image is segmented using a simplified symbolic distance function to obtain segmented regions. A probability model of the region to which any pixel belongs within the segmented region is calculated. The segmented regions include foreground and background regions.

[0119] Calculate the composite energy function of the real-time acquired image, which includes contour energy and texture energy, and update the 3D pose based on the composite energy function;

[0120] Extract the final 3D pose output, which includes the final 3D position and pose of the currently acquired image.

[0121] It should be noted that, in this embodiment, the present invention proposes two core improvements based on the traditional algorithm, namely, simplifying the level set function and the composite energy function. First, the initial image of the target object (high-speed moving target) and the real-time acquired image are obtained. Both the initial image and the real-time acquired image are captured by an industrial camera. The initial image is used to obtain the projection contour and the symbolic distance function as the initial state, while the real-time acquired image is used to update the pose in real time based on the initial state. The specific details will be described in detail in the subsequent specification.

[0122] According to an embodiment of the present invention, obtaining the initial image of the target object specifically includes:

[0123] When the system starts, it acquires the original image and initial pose of the target object;

[0124] The original image is format-converted, and an image pyramid is constructed based on the format-converted original image;

[0125] The projected contour and the simplified symbolic distance function graph are obtained by combining the three-dimensional mesh model of the target object and the initial pose for contour rendering and symbolic distance function simplification.

[0126] It should be noted that, in this embodiment, when the system starts, it acquires the original image of the target object and directly transmits the image data to the GPU global memory through high-speed interfaces such as the PCIe bus, avoiding the CPU copy bottleneck. The original image is then converted to a different format, for example, converting the Bayer array image to an RGB image. Different resolution versions of the original image (e.g., 1:1, 1:2, 1:4) are generated to form an image pyramid structure. The top layer has a low resolution for quick, rough estimation, while the bottom layer has a high resolution for fine optimization. In practical applications, an industrial camera can be used to acquire images at a frame rate of 425. The acquired image data is directly transmitted to the GPU global memory via a preset onboard bus for preprocessing operations such as de-mosaicing and image pyramid construction. Furthermore, the initial 3D pose of the target (6 degrees of freedom: X, Y, Z, rotation Rx, Ry, Rz) is provided by the user or an external system at system startup. This initial 3D pose is used to generate the rendering contour of the first frame, serving as the initial pose for the current frame.

[0127] According to an embodiment of the present invention, contour rendering and simplification of the symbolic distance function are specifically included:

[0128] The three-dimensional mesh model is projected onto a virtual image plane according to the initial pose to obtain the projected contour.

[0129] Each pixel is judged based on the projection contour. If the pixel is within the projection contour, the symbolic distance function value is the first value; otherwise, it is the second value.

[0130] It should be noted that, in this embodiment, the traditional PWP3D algorithm uses an analytical level set function, which has high computational complexity and is difficult to parallelize. Therefore, this invention proposes a simplified Signed Distance Function (SDF), defined as follows:

[0131]

[0132] in, and The adjustable parameters represent the SDF values ​​of the foreground and background regions, respectively. This simplified SDF no longer precisely calculates the distance from each pixel to the contour, but instead assigns fixed negative and positive values ​​to the entire foreground and background regions, respectively. In one embodiment, the first value is a negative value "-1" and the second value is a positive value "+1". Although this simplification introduces some quantization error, in high frame rate image sequences, the impact of this error on the optimization process is negligible because the target pose changes little between adjacent frames. Furthermore, since this SDF calculation process is fully parallelized, it can fully utilize the parallel computing power of the GPU, significantly improving processing speed.

[0133] Specifically, in practical applications, the OpenGL / DirectX graphics API can be used to project the 3D model onto a virtual image plane according to its pose, generating its accurate projection contour and depth map. Based on the rendered contour, each pixel in the image is judged. If the pixel is within the projection contour, the signed distance function value is the first value; otherwise, it is the second value.

[0134] According to an embodiment of the present invention, the real-time acquired image is segmented using a simplified symbolic distance function to obtain segmented regions, and a probability model of the region to which any pixel belongs within the segmented region is calculated, specifically including:

[0135] The foreground and background regions of the real-time acquired image are obtained by segmenting the image using a simplified symbolic distance function.

[0136] Calculate statistical histograms of the color values ​​of all pixels in the foreground and background regions, respectively.

[0137] The probability model of the region to which any pixel belongs is calculated based on the statistical histogram, wherein the probability model includes the foreground probability of any pixel belonging to the foreground region and the background probability of any pixel belonging to the background region.

[0138] It should be noted that, in this embodiment, based on the pixel information of the current image, a statistical histogram model of the foreground and background regions is constructed, and the probability of each pixel belonging to the foreground or background is calculated. Specifically, the real-time acquired image is clearly divided into a foreground region (target object) and a background region using a simplified signed distance function, and statistical histograms of all pixel color values ​​(such as RGB or grayscale) in the foreground and background regions are calculated respectively. Based on the histograms, the probability of any pixel value belonging to the foreground region and the probability of belonging to the background region are calculated.

[0139] According to an embodiment of the present invention, the composite energy of the real-time acquired image is calculated, the composite energy including contour energy and texture energy, and the three-dimensional pose is updated based on the composite energy function, specifically including:

[0140] The contour energy is obtained by measuring the consistency between the projected contour and the real image segmentation based on the probability model and the simplified symbolic distance function graph.

[0141] Within the projected contour region, the sum of squared texture differences between the corresponding regions of the current frame image and the previous frame image is calculated to obtain the texture energy.

[0142] The composite energy is obtained by proportionally summing the contour energy and the texture energy using a preset composite energy function;

[0143] The target matrix of the composite energy function with respect to the pose parameters is calculated using the Gauss-Newton method, and the update amount of the target matrix is ​​solved to update the three-dimensional pose.

[0144] It should be noted that, in this embodiment, in order to enhance the robustness of the algorithm to partial occlusion, the present invention introduces a texture energy term on the basis of traditional contour energy, and constructs the following composite energy function:

[0145] ;

[0146] in, For pixels, For position, Contour energy, defined in the same way as traditional PWP3D, is used to measure the consistency between the projected contour and the real contour. Texture energy is defined as the sum of squares of texture differences within the target region between adjacent frames, and is calculated as follows:

[0147] ;

[0148] in, For pixels, For position, and They represent the first Frame and the Texture information of the target region in the frame image. This is a Heaviside function used to confine texture differences within the target contour area; This is the weighting coefficient for texture energy, adjusted according to the texture richness of the target object: for objects with rich textures, take... For objects with fewer textures, take .

[0149] Furthermore, the composite energy function considers both contour consistency and texture consistency during the optimization process, ensuring that the system can still perform stable tracking based on texture information even when the target contour is partially occluded. Specifically, in the iterative optimization, the Gauss-Newton method is used to calculate the composite energy function for the pose parameters. The target matrix is ​​the Jacobian matrix, and the update amount of the target matrix is ​​solved. To update and obtain the three-dimensional pose This process proceeds from coarse to fine on the image pyramid, first converging rapidly at the low-resolution layer, and then using the result as the initial value for the high-resolution layer for fine optimization.

[0150] Furthermore, the three-dimensional pose is finally extracted and output. The output results include the final three-dimensional position and orientation of the currently acquired image. Specifically, the pose data (usually a 4x4 transformation matrix or a 6-dimensional vector) is transferred from the GPU memory to the control unit (CPU) for output. This output is the final result of the entire algorithm process and is directly transmitted to the active gimbal system or robot for real-time closed-loop control.

[0151] A third aspect of the present invention provides a computer-readable storage medium comprising a monocular vision three-dimensional target tracking method program, wherein when executed by a processor, the monocular vision three-dimensional target tracking method program implements the steps of a monocular vision three-dimensional target tracking method as described in any of the preceding claims.

[0152] Figure 3 A block diagram of a monocular vision three-dimensional target tracking device according to the present invention is shown.

[0153] like Figure 3 As shown, a fourth aspect of the present invention provides a monocular vision three-dimensional target tracking device, applied to any of the monocular vision three-dimensional target tracking methods described in any one of the claims, wherein the device comprises:

[0154] The system includes a high-speed monocular RGB camera, an image processing unit, a control unit, and an active gimbal system.

[0155] The high-speed monocular RGB camera includes an industrial camera used to acquire two-dimensional image sequences of high-speed moving target objects;

[0156] The image processing unit is used to calculate the pose information of the high-speed moving target object from the two-dimensional image sequence using a preset algorithm;

[0157] The control unit is used to calculate the adjustment angle of the multi-degree-of-freedom gimbal based on the pose information and output control commands.

[0158] The active gimbal system includes a multi-degree-of-freedom gimbal controlled by a drive motor, used to support the industrial camera and adjust the shooting angle of the industrial camera based on the control commands.

[0159] It should be noted that, in this embodiment, the high-speed monocular RGB camera is a high frame rate industrial camera (such as XIMEA MC031CG-SY), with a frame rate of up to "425Hz" and an image resolution of "512×512" pixels, used to acquire continuous image sequences of the target object. The image processing unit is a computer equipped with a high-performance GPU (such as one equipped with an NVIDIA Titan Black graphics card), responsible for running the Fast-PWP3D algorithm proposed in this invention to achieve real-time estimation of the target's three-dimensional pose.

[0160] Furthermore, the control unit is used to receive the pose information output by the vision system, generate control commands to drive the gimbal or robot actuator, and realize closed-loop control. In the active gimbal system, the camera is mounted on a two-degree-of-freedom (pitch and yaw) active gimbal, which is controlled by a direct drive motor and has high dynamic response capability. It can realize rapid adjustment of the camera's field of view to track the target. The specific processes of the image processing unit and the control unit have been described in the above embodiments and will not be repeated in this embodiment.

[0161] This invention discloses a monocular vision 3D target tracking method, system, storage medium, and device, which effectively solves the 3D tracking problem of monocular vision in high-speed, occluded, and dynamic scenes. It has advantages such as high real-time performance, strong robustness, and high integration, and is suitable for various industrial vision control scenarios.

[0162] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0163] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0164] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0165] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0166] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

Claims

1. A monocular vision-based three-dimensional target tracking method, characterized in that, The method includes the following steps: Acquire an initial image and a real-time acquired image of the target object, wherein the initial image includes a projected contour and a simplified symbolic distance function graph; The real-time acquired image is segmented using a simplified symbolic distance function to obtain segmented regions. A probability model of the region to which any pixel belongs within the segmented region is calculated. The segmented regions include foreground and background regions. The composite energy of the real-time acquired image is calculated, which includes contour energy and texture energy, and the 3D pose is updated based on the composite energy function. The final 3D pose is extracted and output, including the final 3D position and pose of the currently acquired image.

2. The monocular vision three-dimensional target tracking method according to claim 1, characterized in that, The acquisition of the initial image of the target object specifically includes: When the system starts, it acquires the original image and initial pose of the target object; The original image is format-converted, and an image pyramid is constructed based on the format-converted original image; The projected contour and the simplified symbolic distance function graph are obtained by combining the three-dimensional mesh model of the target object and the initial pose for contour rendering and symbolic distance function simplification.

3. The monocular vision three-dimensional target tracking method according to claim 2, characterized in that, Contour rendering and simplification of the signed distance function are performed, specifically including: The three-dimensional mesh model is projected onto a virtual image plane according to the initial pose to obtain the projected contour. Each pixel is judged based on the projection contour. If the pixel is within the projection contour, the symbolic distance function value is the first value; otherwise, it is the second value.

4. The monocular vision three-dimensional target tracking method according to claim 3, characterized in that, The real-time acquired image is segmented using a simplified signed distance function to obtain segmented regions. A probability model of the region to which any pixel within a segmented region belongs is calculated, specifically including: The foreground and background regions of the real-time acquired image are obtained by segmenting the image using a simplified symbolic distance function. Calculate statistical histograms of the color values ​​of all pixels in the foreground and background regions, respectively. The probability model of the region to which any pixel belongs is calculated based on the statistical histogram, wherein the probability model includes the foreground probability of any pixel belonging to the foreground region and the background probability of any pixel belonging to the background region.

5. The monocular vision three-dimensional target tracking method according to claim 4, characterized in that, Calculate the composite energy of the real-time acquired image, which includes contour energy and texture energy, and update the 3D pose based on the composite energy function, specifically including: The contour energy is obtained by measuring the consistency between the projected contour and the real image segmentation based on the probability model and the simplified symbolic distance function graph. Within the projected contour region, the sum of squared texture differences between the corresponding regions of the current frame image and the previous frame image is calculated to obtain the texture energy; The composite energy is obtained by proportionally summing the contour energy and the texture energy using a preset composite energy function; The target matrix of the composite energy function with respect to the pose parameters is calculated using the Gauss-Newton method, and the update amount of the target matrix is ​​solved to update the three-dimensional pose.

6. A monocular vision three-dimensional target tracking system, characterized in that, The system includes a memory and a processor. The memory contains a monocular vision 3D target tracking method program, which, when executed by the processor, performs the following steps: Acquire an initial image and a real-time acquired image of the target object, wherein the initial image includes a projected contour and a simplified symbolic distance function graph; The real-time acquired image is segmented using a simplified symbolic distance function to obtain segmented regions. A probability model of the region to which any pixel belongs within the segmented region is calculated. The segmented regions include foreground and background regions. Calculate the composite energy function of the real-time acquired image, which includes contour energy and texture energy, and update the 3D pose based on the composite energy function; The final 3D pose is extracted and output, including the final 3D position and pose of the currently acquired image.

7. A monocular vision three-dimensional target tracking system according to claim 6, characterized in that, The acquisition of the initial image of the target object specifically includes: When the system starts, it acquires the original image and initial pose of the target object; The original image is format-converted, and an image pyramid is constructed based on the format-converted original image; The projected contour and the simplified symbolic distance function graph are obtained by combining the three-dimensional mesh model of the target object and the initial pose for contour rendering and symbolic distance function simplification.

8. The monocular vision three-dimensional target tracking system according to claim 7, characterized in that, Contour rendering and simplification of the signed distance function are performed, specifically including: The three-dimensional mesh model is projected onto a virtual image plane according to the initial pose to obtain the projected contour. Each pixel is judged based on the projection contour. If the pixel is within the projection contour, the symbolic distance function value is the first value; otherwise, it is the second value.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a monocular vision three-dimensional target tracking method program, which, when executed by a processor, implements the steps of a monocular vision three-dimensional target tracking method as described in any one of claims 1 to 6.

10. A monocular vision three-dimensional target tracking device, characterized in that, An apparatus for use in a monocular vision 3D target tracking method as described in any one of claims 1 to 6, wherein the apparatus comprises: The system includes a high-speed monocular RGB camera, an image processing unit, a control unit, and an active gimbal system. The high-speed monocular RGB camera includes an industrial camera used to acquire two-dimensional image sequences of high-speed moving target objects; The image processing unit is used to calculate the pose information of the high-speed moving target object from the two-dimensional image sequence using a preset algorithm; The control unit is used to calculate the adjustment angle of the multi-degree-of-freedom gimbal based on the pose information and output control commands. The active gimbal system includes a multi-degree-of-freedom gimbal controlled by a drive motor, used to support the industrial camera and adjust the shooting angle of the industrial camera based on the control commands.

Images