A method for tracking moving objects based on implicit neural networks
By establishing a continuous function mapping through an implicit neural network, the problem of pixel coordinate changes caused by object motion in 3D measurement of grating projection is solved, realizing efficient and accurate object tracking, which is applicable to reverse engineering, industrial inspection and oral medicine and other fields.
Patent Information
- Application Number
- CN202410441923.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-12
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2044-04-12
AI Technical Summary
In 3D measurement using grating projection, changes in pixel coordinates caused by moving objects lead to measurement errors, and existing technologies struggle to effectively track object positions to eliminate these motion errors.
By employing an implicit neural network representation method, driven by perspective geometry imaging and physical knowledge, a continuous and differentiable function mapping is established. The expressive power of the implicit neural network is used to quickly obtain the changes in the motion of an object, thereby achieving the tracking of moving objects.
It achieves fast and high-precision tracking of moving objects, eliminating pixel coordinate changes caused by object motion in the phase-shift method, and meeting the needs of rapid measurement on actual production lines.
Smart Images

Figure CN118397041B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of digital grating projection 3D measurement technology, and in particular relates to a method for tracking moving objects based on implicit neural networks. Background Technology
[0002] Grating projection 3D measurement technology is an active, non-contact 3D measurement technique characterized by low cost, high measurement speed, large field of view, and high resolution. It is widely used in reverse engineering, industrial inspection, dentistry, and cultural relic restoration. A typical grating projection 3D measurement system includes a camera and a projector. The projector projects fringes containing phase information onto the surface of the object being measured. The fringes are highly modulated and distorted by the surface of the object. The camera captures an image of the object with the distorted fringes, and then performs phase extraction to obtain the pixel coordinates of each point on the object's surface in the projector's image coordinate system. Finally, the 3D coordinates of each point on the object's surface in the world coordinate system are calculated based on the principle of triangulation. The phase can be calculated using the phase-shifting method. The phase-shifting method solves for the phase in the time domain, does not rely on spatial texture information, and has a certain ability to overcome differences in surface texture and ambient light interference, exhibiting strong robustness. During measurement, the phase-shifted sinusoidal fringe projection method requires at least three sinusoidal fringe patterns with known phase shifts to be projected and acquired on the surface of the object being measured, provided that the object remains stationary. This is because the movement of the object will cause the phase encoding information of the measured point on the projection plane of the projector and the pixel position on the imaging plane of the camera to change simultaneously. If the traditional phase-shifting method is used, motion errors will occur, so it is necessary to track the position of the target to eliminate motion errors. Summary of the Invention
[0003] For tracking moving objects, a moving object tracking method based on implicit neural networks is proposed.
[0004] Because the movement of an object causes changes in the position of the measured point on the camera image, and the complex texture and high reflectivity of the object's surface result in significant differences in the reflectivity of different regions to the sinusoidal fringe structured light signal, it is essential to track the same measured point on the camera image during object movement; otherwise, substantial measurement errors will occur. Utilizing the physical mechanism of fringe pattern formation and the texture information of the object's surface, and driven by the physical principles of perspective geometry imaging and measurement, an implicit neural network representation method is employed to describe the fringe pattern carrying texture information. The differentiability property of the neural network is used to quickly obtain the changes in object motion, ultimately achieving the tracking of the moving object, i.e., the mapping relationship between pixel coordinates before and after motion.
[0005] The technical solution adopted in this invention is as follows:
[0006] A method for tracking moving objects based on implicit neural networks, comprising the following steps:
[0007] Step 1: Based on the known camera parameters, namely the lens distortion coefficient, perform distortion correction on the pixel coordinates;
[0008] Step 2: Based on the camera calibration results and the world coordinate system O w x w y w z w With camera coordinate system O c x c y c z c The relative pose relationship, based on the physical knowledge of perspective geometry imaging, is used to determine the distortion-corrected image coordinates (u). c ,v c Mapped to the physical space coordinates (x) of the object surface w ,y w );
[0009] Based on the principle of pinhole imaging, any point (x) on the surface of an object before and after motion... w ,y w The pixel position correspondence in the camera image is as follows:
[0010]
[0011] In equation (4), (u c ,v c ) and (u c ',v c ') represent the points (x, y) on the object's surface before and after the motion. w ,y w K corresponds to the pixel coordinates of the camera image. c For camera intrinsic parameters, R c T c Let e1, e2, and e3 be the rotation and translation matrices between the camera coordinate system and the world coordinate system before motion, respectively. w x w Shaft, O w y w Shaft and O w z w The unit column vectors of the axes in the camera coordinate system, where s, s', and s" are scaling factors; R c′ T c′Let e1', e2', and e3' represent the rotation and translation matrices after the motion, respectively. Let e1', e2', and e3' be the unit column vectors of each coordinate axis of the world coordinate system in the camera coordinate system after the motion. Let [Δe1, Δe2, Δe3] be the rotation matrix of the measured object before and after the motion in the world coordinate system. Let α, β, and γ be the rotation angles around the x-axis, y-axis, and z-axis of the world coordinate system, respectively. Let Δx, Δy, and Δz be the displacements of the object on the x-axis, y-axis, and z-axis of the world coordinate system, respectively.
[0012] According to equation (4), motion tracking of surface texture within the camera coordinate system before and after motion is achieved by mapping the physical space coordinates of the object's surface before and after motion to the camera image coordinates; based on the pre-calibration results of the camera, projector, and the object being measured, the relative relationship between the world coordinate system and the camera coordinate system is known, i.e., the aforementioned parameter K c T c e1, e2, and e3 are known quantities, while α, β, γ, Δx, Δy, and Δz caused by the object's motion are unknown parameters. If these six unknown parameters can be determined, the motion-induced (u) in the camera image... c ,v c )→(p k ,q k The mapping relationship (p) k ,q k )=f k (u c ,v c This can be determined by )
[0013] Step 3: Using an implicit neural network in physical space (x) w ,y w Establish a continuous and differentiable A(x) on ) w ,y w B(x) w ,y w ) and Φ(x w ,y w The function mapping representation of ) uses the six unknown parameters caused by the object's motion as implicit neural network parameters and employs gradient descent to solve them quickly; the input of the implicit neural network is the physical space coordinates (x, y, z) with the six unknown parameters. w ,y w Before input, the physical space coordinates need to be normalized. Then, using the expressive power of the implicit neural network, the physical space points (x, y) on the object's surface are obtained. w ,y w Texture and height feature representation parameters: A(x) w ,y w B(x) w ,y w ) and Φ(x w ,yw Based on this, the calculation results of the stripe pattern are given by combining the measurement step information of the stripe pattern. The residual is obtained by comparing it with the stripe pattern acquired by the camera. The residual is used as the optimization target to update and iterate the six unknown parameters caused by the object motion of the stripe pattern and the implicit neural network weight parameters.
[0014] The activation function of the implicit neural network is Sin(·), and the parameter A(x) represents the surface texture and height features of the object. w ,y w B(x) w ,y w ) and Φ(x w ,y w Strong ability to express details;
[0015] Step 4: Substitute the 6 unknown parameters obtained in Step 3 into formula (4) to obtain the x-axis values of the object's surface before and after the motion. w ,y w ) Corresponding camera image pixel coordinates (u c ,v c ) and (u c ',v c The mapping relationship enables the tracking of coordinate changes caused by the movement of an object.
[0016] The beneficial effects of this invention are as follows: This invention is a moving object tracking method based on implicit neural networks, which can achieve fast and high-precision coordinate tracking, solve the pixel coordinate changes caused by object motion in the phase-shift method, and meet the needs of rapid measurement on actual production lines. Attached Figure Description
[0017] Figure 1 Physics-driven object tracking method using implicit neural network representations Detailed Implementation
[0018] Considering the motion of the object, according to the N-step phase-shift sinusoidal fringe projection measurement method, at the k-th step, the projector projects the k-th sinusoidal fringe pattern. Correspondingly, the grayscale of the measured point on the object surface in the image acquired by the camera is:
[0019] I k (p k ,q k ) = A k (p k ,q k )+B k (p k ,q k cos(Φ) k (p k ,q k )+2(k-1)π / N) (1)
[0020] In equation (1), (p k ,q k ) represents the pixel position of the same measured point in the stripe image acquired by the camera at step k. k (p k ,q k A represents the grayscale value of the measured point at step k in the fringe image. k (p k ,q k ) and B k (p k ,q k ) represents the parameter related to ambient light and surface reflectivity at step k of the measured point, Φ k (p k ,q k ) represents the phase of the sinusoidal stripe encoding projected by the projector at that point in step k.
[0021] Considering the general motion of the object, the amplitude of the object's motion is not large during the entire N-step phase shift measurement, due to A k (p k ,q k ) and B k (p k ,q k A represents a parameter related to ambient light and surface reflectivity at the same measured point. k (p k ,q k ) and B k (p k ,q k The phase Φ of the sinusoidal fringe remains unchanged. Based on the solution for eliminating phase error proposed above, the phase Φ of the sinusoidal fringe remains unchanged. k (p k ,q k The three unknowns A also remain unchanged. k (p k ,q k B k (p k ,q k ) and Φ k (p k ,q k All of them remain unchanged during N-step phase shift measurements:
[0022]
[0023] In equation (2), (u c ,v c p1 represents the pixel position of that point in the camera's stripe image at step 1, i.e., p1 = u c,q1=v c .
[0024] According to equation (2), equation (1) becomes:
[0025] I k (p k ,q k )=A(u c ,v c )+B(u c ,v c )cos(Φ(u c ,v c (3) + 2(k-1)π / N
[0026] Equation (3) shows that for a certain measured point (u) c ,v c There are only 3 unknowns: A(u) c ,v c ), B(u) c ,v c ) and Φ(u c ,v c The solution can be obtained using at least three fringe patterns (N≥3). However, it is necessary to track the pixel position changes of the measured point on the camera image to find (u c ,v c )→(p k ,q k The mapping relationship of (p) k ,q k )=f k (u c ,v c ), using pixel position (p k ,q k Determine the gray value I on the k-th fringe pattern. k (p k ,q k Then, according to equation (3), the phase encoding information Φ(u) is obtained. c ,v c ).
[0027] To track coordinate changes caused by motion, considering that the motion direction of objects on the actual production line is singular and fixed, the experiment sets the motion direction to the x-axis of the world coordinate system. The object being tested is a cast billet. The specific implementation is as follows:
[0028] Step 1: Adjust the projection angle of the projector so that the projected stripes and equiphase lines are parallel to the direction of the object's movement. The camera captures an image with stripe information. The camera resolution is 1080*1440, and the projector resolution is 720*1280.
[0029] Step 2: Perform system calibration and obtain camera intrinsic parameters (K). c ), extrinsic parameters (rotation matrix R) c Translation matrix T c ) and distortion coefficient (D c );
[0030] Step 3: Based on the distortion model and system calibration, obtain the camera distortion coefficients (D). c Including k1, k2, p1, p2), distortion correction is performed on the pixel coordinates of the image; the distortion model is:
[0031]
[0032] Where k1 and k2 are radial distortion parameters, p1 and p2 are tangential distortion parameters, (u d ,v d ) represents the distorted coordinates, (u c ,v c () represents the undistorted coordinates.
[0033] Step 4: Based on the camera intrinsic and extrinsic parameters (K) obtained from system calibration c R c =[e1,e2,e3]、T c ), for the distortion-corrected pixels (u c ,v c The coordinates are transformed and mapped to the physical space coordinates (x, y) of the billet surface. w ,y w ), because the object is in O w x w y w Moving along the x-axis in a plane, i.e., f(α,β,γ)=1, z w =0, Δy=0, Δz=0;
[0034]
[0035] Step 5: Normalize the physical space coordinates and the grayscale values of the image, mapping them to the [-1,1] interval;
[0036] Step 6: Input the physical space coordinates with the unknown displacement Δx into the implicit neural network for training. Register the displacement Δx as a trainable neural network parameter and optimize iteratively together with the internal parameters of the neural network.
[0037] Step 7: The activation function of the implicit neural network is Sin(·), which represents the surface texture and height features of the billet as parameters A(x). w ,y w B(x) w ,y w) and Φ(x w ,y w It has strong ability to express details. The neural network is a fully connected neural network with 3 hidden layers. Each layer has 512 neurons. It uses the Adam optimizer, with a learning rate of lr = 1e-4 and a training iteration count of 2000.
[0038] Step 8: The input to the implicit neural network is the normalized physical space coordinates (x, y) with the unknown displacement Δx. w ,y w The output is a physical space point (x). w ,y w The texture and height feature representation parameter A(x) w ,y w B(x) w ,y w ) and Φ(x w ,y w Substitute the output into the grayscale function and subtract it from the grayscale of the actual acquired image. Use the residual as the optimization target to iteratively update the motion displacement parameters and the internal parameters of the neural network. The training and optimization of the model are carried out alternately: (1) The motion displacement Δx of the billet is quickly calculated, that is, the surface texture of the moving billet is quickly tracked. At this time, the weight parameters of the neural network model are frozen, and the existing expressive ability of the neural network is used to quickly predict the motion displacement of the billet in the current striped image. Since there is only Δx k To optimize parameters, the calculation speed is fast and the efficiency is high. (2) Neural network weight update, updating the descriptive ability of the neural network. The neural network weights are updated using several stripe images that were not used in the training of the latest neural network weight update, as well as the corresponding measurement steps and billet movement displacement information. Since the billet movement displacement is small, the overlap ratio between the newly trained stripe images and the previously trained stripe images in the camera's field of view is very large. The update of the neural network weights only needs to update the descriptive ability of a small part of the billet surface that has newly entered the camera's field of view due to the billet movement, and the convergence speed is fast. These two methods are performed alternately, which is efficient and converges quickly.
[0039] Step 9: Substitute the displacement Δx of the billet obtained from the implicit neural network training into formula (4) to obtain the point (x) on the surface of the billet before and after the movement. w ,y w ) Corresponding camera image pixel coordinates (u c ,v c ) and (u c ',v c The mapping relationship enables the tracking of coordinate changes caused by the movement of an object.
Claims
1. A method for tracking moving objects based on implicit neural networks, characterized in that, The steps are as follows: Step 1: Based on the known camera parameters, namely the lens distortion coefficient, perform distortion correction on the pixel coordinates; Step 2: Based on the camera calibration results and the world coordinate system O w x w y w z w With camera coordinate system O c x c y c z c The relative pose relationship, based on the physical knowledge of perspective geometry imaging, is used to determine the distortion-corrected image coordinates (u). c , v c Mapped to the physical space coordinates (x) of the object surface w , y w ); Based on the principle of pinhole imaging, any point (x) on the surface of an object before and after motion... w , y w The pixel position correspondence in the camera image is as follows: (4) In equation (4), (u c , v c ) and (u c ', v c ') represent the points (x, y) on the object's surface before and after the motion. w , y w K corresponds to the pixel coordinates of the camera image. c For camera intrinsic parameters, R c T c Let e1, e2, and e3 be the rotation and translation matrices between the camera coordinate system and the world coordinate system before motion, respectively. w x w Shaft, O w y w Shaft and O w z w The unit column vectors of axes in the camera coordinate system, where s, s', and s'' are scaling factors; , Let e1', e2', and e3' represent the rotation and translation matrices after the motion, respectively. Let e1', e2', and e3' be the unit column vectors of each coordinate axis in the world coordinate system in the camera coordinate system after the motion. Let [Δe1, Δe2, Δe3] be the rotation matrix of the measured object before and after the motion in the world coordinate system. Let α, β, and γ be the rotation angles around the x-axis, y-axis, and z-axis in the world coordinate system, respectively. Let Δx, Δy, and Δz be the displacements of the object on the x-axis, y-axis, and z-axis in the world coordinate system, respectively. According to equation (4), motion tracking of surface texture within the camera coordinate system before and after motion is achieved by mapping the physical space coordinates of the object's surface before and after motion to the camera image coordinates; based on the pre-calibration results of the camera, projector, and the object being measured, the relative relationship between the world coordinate system and the camera coordinate system is known, i.e., the aforementioned parameter K c T c e1, e2, and e3 are known quantities, while α, β, γ, Δx, Δy, and Δz caused by the object's motion are unknown parameters. If these six unknown parameters can be determined, the motion-induced (u) in the camera image... c , v c )→(p k , q k The mapping relationship (p) k , q k ) = f k (u c , v c This can be determined by ) Step 3: Using an implicit neural network in physical space (x) w , y w Establish a continuous and differentiable A(x) on ) w , y w B(x) w ,y w ) and Φ(x w , y w The function mapping representation of ) uses the six unknown parameters caused by the object's motion as implicit neural network parameters and employs gradient descent to solve them quickly; the input of the implicit neural network is the physical space coordinates (x, y, z) with the six unknown parameters. w , y w Before input, the physical space coordinates need to be normalized. Then, using the expressive power of the implicit neural network, the physical space points (x, y) on the object's surface are obtained. w , y w Texture and height feature representation parameters: A(x) w , y w B(x) w , y w ) and Φ(x w , y w Based on this, the calculation results of the stripe pattern are given by combining the measurement step information of the stripe pattern. The residual is obtained by comparing it with the stripe pattern acquired by the camera. The residual is used as the optimization target to update and iterate the six unknown parameters caused by the object motion of the stripe pattern and the implicit neural network weight parameters. The activation function of the implicit neural network is Sin(·), and the parameter A(x) represents the surface texture and height features of the object. w , y w B(x) w , y w ) and Φ(x w , y w Strong ability to express details; Step 4: Substitute the 6 unknown parameters obtained in Step 3 into formula (4) to obtain the x-axis values of the object surface before and after the motion. w ,y w ) Corresponding camera image pixel coordinates (u c , v c ) and (u c ', v c The mapping relationship enables the tracking of coordinate changes caused by the movement of an object.
2. The moving object tracking method based on an implicit neural network according to claim 1, characterized in that, After camera calibration, the camera intrinsic parameter K can be obtained. c Extrinsic parameter: rotation matrix R c Translation matrix T c and distortion coefficient D c .
3. The moving object tracking method based on an implicit neural network according to claim 2, characterized in that, Based on the camera distortion coefficients obtained from the distortion model and system calibration, D c Including k1, k2, p1, and p2, distortion correction is performed on the pixel coordinates of the image; the distortion model is: Where k1 and k2 are radial distortion parameters, p1 and p2 are tangential distortion parameters, (u d , v d ) represents the distorted coordinates, (u c , v c () represents the undistorted coordinates. .
4. A method for tracking moving objects based on an implicit neural network according to claim 1, 2, or 3, characterized in that, The training and optimization of the implicit neural network are carried out alternately: (1) The displacement Δx of the object is quickly calculated, that is, the surface texture of the moving object is quickly tracked; at this time, the weight parameters of the implicit neural network model are frozen, and the existing expressive ability of the implicit neural network is used to quickly predict the displacement of the object in the current stripe image. (2) Implicit neural network weight update: update the description capability of the implicit neural network by using the latest stripe images that were not used in the training of the implicit neural network weight update, as well as the corresponding measurement steps and object motion displacement information, to update the implicit neural network weights.
5. The moving object tracking method based on an implicit neural network according to claim 1, characterized in that, The implicit neural network is a fully connected neural network containing 3 hidden layers, each with 512 neurons, and uses the Adam optimizer.
