A scene flow acquisition method based on flight time

Through the scene flow acquisition method based on time of flight, the feature extraction and matching of TOF depth data and infrared image data is used to generate three-dimensional motion vectors, which solves the problem that the existing technology cannot obtain scene flow information in three-dimensional space, and achieves high precision, Real-time three-dimensional fluid motion measurement.

CN118230169BActive Publication Date: 2024-11-26XIHUA UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410441267.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-04-12
Publication Date
2024-11-26
Estimated Expiration
2044-04-12

AI Technical Summary

Technical Problem

The prior art cannot effectively obtain scene flow information in three-dimensional space, and can only obtain information at the measured section, and cannot characterize scene flow information in other parts.

Method used

Using a scene flow acquisition method based on time of flight, three-dimensional motion vectors and velocity vectors are generated through feature extraction of TOF depth data and infrared image data, joint feature descriptor matching, KD tree search and similarity measurement, and sparse scene flow is obtained. The dense scene stream is obtained by upsampling.

Benefits of technology

It realizes real-time, high frame rate, high synchronization and high precision to obtain the entire three-dimensional fluid space, so it can measure the state of three-dimensional fluid motion without contact, and has high parallelism in calculations, which facilitates accelerated computing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118230169B_ABST
    Figure CN118230169B_ABST
Patent Text Reader

Abstract

The invention discloses a scene flow acquisition method based on time of flight, comprising the following steps: acquiring TOF depth data and infrared image data at different times in a target flow space; performing feature extraction on the TOF depth data and infrared image data respectively; combining two feature vectors at the same time to obtain a joint feature descriptor; performing feature matching on the joint feature descriptor through KD tree search and similarity measurement to obtain feature points of the same name and their matching relationship; generating three-dimensional motion vectors and three-dimensional velocity vectors between feature points of the same name that change with time based on the matching relationship of the feature points of the same name, that is, obtaining a sparse scene flow; and performing upsampling on the sparse scene flow to obtain a dense scene flow. The method can obtain the scene flow of the entire three-dimensional fluid space in real time with high frame rate, high synchronization and high precision.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to the field of scene flow calculation of transparent flow fields, and in particular to a scene flow acquisition method based on flight time. Background Art

[0002] Optical flow and scene flow acquisition methods are fundamental and key issues in the fields of computer vision and fluid mechanics. Current optical flow acquisition technology is mainly based on two-dimensional image processing methods, including particle image velocimetry (PIV) and particle trajectory velocimetry (PSV). Two-dimensional image processing methods can only obtain the fluid motion state of a cut or cross section of the scene being measured, and cannot obtain the complete motion state in three-dimensional space.

[0003] Scene flow acquisition methods mostly use three-dimensional PIV, light field camera or binocular PIV methods, but these methods obtain the optical flow of multiple two-dimensional sections, and then generate three-dimensional scene flow through synthesis processing. The scene flow information only has information at the measured section and cannot represent the scene flow information of other parts of the three-dimensional scene. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the present invention provides a scene flow acquisition method based on time of flight, which solves the problem that the existing scene flow acquisition method cannot characterize the scene flow information of other parts of the three-dimensional scene.

[0005] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is:

[0006] A method for acquiring scene flow based on time of flight is provided, which comprises the following steps:

[0007] S1, obtaining TOF depth data and infrared image data at different times in the target flow space;

[0008] S2, extracting features from the TOF depth data and the infrared image data respectively, and obtaining feature vectors of the feature points of the tracer particles in the target flow space;

[0009] S3, combining the two feature vectors at the same time to obtain a joint feature descriptor;

[0010] S4, through KD tree search and similarity measurement, feature matching is performed on the joint feature descriptors to obtain feature points with the same name and their matching relationships;

[0011] S5, generating a three-dimensional motion vector and a three-dimensional velocity vector between the feature points with the same name that change over time based on the matching relationship between the feature points with the same name, that is, obtaining a sparse scene flow;

[0012] S6. Upsample the sparse scene flow to obtain a dense scene flow.

[0013] Furthermore, the specific method of step S1 is:

[0014] The TOF device emits a modulated light pulse signal in the infrared frequency band, and receives the modulated light pulse signal reflected by the target flow space and the tracer particles therein, thereby obtaining the TOF depth data and infrared image data at the corresponding moment.

[0015] Furthermore, the specific method for extracting features from TOF depth data in step S2 is:

[0016] Based on the reflectivity difference between the tracer particles and the target flow space, a convolutional neural network is used to extract the spatial position of the tracer particles in the TOF depth data to obtain the three-dimensional features of the TOF depth data, that is, to obtain the feature vector of the tracer particle feature points corresponding to the TOF depth data in the target flow space.

[0017] Furthermore, the specific method for extracting features from infrared image data in step S2 is:

[0018] Feature extraction operators are used to extract features from infrared image data to obtain two-dimensional features of the infrared image data, that is, feature vectors of tracer particle feature points corresponding to the infrared image data in the target flow space are obtained; the feature extraction operators include SIFT operators, SURF operators and ORB operators; the two-dimensional features of the infrared image data include the image coordinates of the feature points and the feature descriptors of the feature points.

[0019] Furthermore, the specific method of step S3 is:

[0020] The two feature vectors at the same time are normalized respectively, and the two normalized feature vectors at the same time are combined into a complex vector to obtain a joint feature descriptor.

[0021] Furthermore, the specific method of step S4 includes the following sub-steps:

[0022] S4-1, taking the target feature point in the joint feature descriptor corresponding to time T0 as a reference, searching the feature point closest to the target feature point in the joint feature descriptor corresponding to time T1 through the KD tree to obtain a feature point pair; wherein time T0 and time T1 are adjacent sampling times;

[0023] S4-2, according to the formula:

[0024]

[0025] Get the joint feature similarity between time T0 and time T1 in is the feature descriptor of the cth feature point at time T0; is the feature descriptor of the cth feature point at time T1; N is the total number of feature points in the TOF depth data; M is the total number of feature points in the infrared image data;

[0026] S4-3, recording the feature point pairs whose joint feature similarity value is greater than the similarity threshold as feature points with the same name;

[0027] S4-4, through the random sampling consensus method, the feature points with the same name are divided into internal points and external points, and the corresponding relationship between the feature points with the same name with the smallest internal point error is taken as the time T0 and T1 moment The matching relationship between them.

[0028] Furthermore, the specific method of step S5 is:

[0029] According to the matching relationship of feature points with the same name, the three-dimensional motion vector and three-dimensional velocity vector that change with time between feature points with the same name are generated; the three-dimensional motion vector is used to describe the coordinate change of the corresponding feature point in the three-dimensional space; the three-dimensional velocity vector is used to describe the velocity of the corresponding feature point in the three-dimensional space, that is, the sparse scene flow in the target flow space is obtained.

[0030] Furthermore, step S6 also includes step S7: rendering, displaying and visualizing the interaction between the sparse scene flow and the dense scene flow; wherein the rendered sparse scene flow is used to track and locate salient targets in the target flow space; and the rendered dense scene flow is used to interpret and understand the target flow space.

[0031] Furthermore, visualization methods include three-dimensional arrow diagrams, three-dimensional velocity diagrams, flow tube diagrams and flow band diagrams; and point cloud animations with different pseudo colors are used in the visualization process to represent changes in the target flow space at different times.

[0032] The beneficial effects of the present invention are:

[0033] 1. This method uses a TOF device to directly shoot the target flow space and the tracer examples therein, and synchronously obtains TOF depth data and infrared image data at high speed, which can obtain the scene flow of the entire three-dimensional fluid space in real time with high frame rate, high synchronization and high precision. This capability is not available in traditional two-dimensional PIV measurement methods, three-dimensional PIV measurement methods, light field camera measurement methods and binocular camera measurement methods.

[0034] 2. This method enables the scene flow of three-dimensional fluid to be measured contactlessly in motion, with flexible imaging posture and no need for prior calibration of the measuring device. It has significant advantages over traditional scene flow acquisition methods that use calibration devices.

[0035] 3. This method uses tracer particles (the density of tracer particles is generally tens of thousands or hundreds of thousands per cubic decimeter) as the visualization medium for the motion of the transparent flow field. Thanks to the independence of each tracer particle, the processing methods are also decoupled from each other, with a high degree of computational parallelism, which facilitates accelerated calculations. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a schematic diagram of the process of this method;

[0037] Figure 2 Schematic diagram of application scenario of the method in the embodiment.

[0038] Among them: 1. TOF device; 2. TOF irradiation unit; 3. TOF imaging sensor; 4. modulated light pulse signal; 5. reflected light; 6. target flow space; 7. tracer particles. DETAILED DESCRIPTION

[0039] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0040] like Figure 1 As shown, the scene flow acquisition method based on flight time includes the following steps:

[0041] S1, obtaining TOF depth data and infrared image data at different times in the target flow space;

[0042] S2, extracting features from the TOF depth data and the infrared image data respectively, and obtaining feature vectors of the feature points of the tracer particles in the target flow space;

[0043] S3, combining the two feature vectors at the same time to obtain a joint feature descriptor;

[0044] S4, through KD tree search and similarity measurement, feature matching is performed on the joint feature descriptors to obtain feature points with the same name and their matching relationships;

[0045] S5, generating a three-dimensional motion vector and a three-dimensional velocity vector between the feature points with the same name that change over time based on the matching relationship between the feature points with the same name, that is, obtaining a sparse scene flow;

[0046] S6. Upsample the sparse scene flow to obtain a dense scene flow.

[0047] In the specific implementation process, Figure 2As shown, the specific method of step S1 is: a modulated light pulse signal 4 in the infrared frequency band is emitted by the TOF irradiation unit 2 on the TOF device 1, and the reflected light 5 reflected by the target flow space 6 and the tracer particles 7 therein is received by the TOF imaging sensor 3, so as to obtain the TOF depth data D(x, y, T0) and infrared image data I(x, y, T0) at time T0, wherein x is the horizontal axis coordinate of the TOF imaging sensor 3, and y is the vertical axis coordinate of the TOF imaging sensor 3. When the time interval ΔT passes and the time T1 is reached, the operation at time T0 is repeated to obtain the OF depth data D(x, y, T1) and infrared image data I(x, y, T1) at time T1. The time interval ΔT is set by the frame rate f and the exposure time ΔT. E Sure: Preferably, ΔT>2ΔT E , to avoid exposure crosstalk between T0 and T1.

[0048] The tracer particles 7 can be made of aluminum powder, polystyrene balls or plexiglass powder. This type of tracer particles has a high reflectivity to the modulated light pulses of the wavelength emitted by the TOF irradiation unit, and the fluid in the measured fluid space has a high absorption rate to the modulated light pulses of the wavelength emitted by the TOF irradiation unit.

[0049] The specific method for extracting features from the TOF depth data in step S2 is: based on the reflectivity difference between the tracer particles and the target flow space, a convolutional neural network is used to extract the spatial position of the tracer particles in the TOF depth data to obtain the three-dimensional features F of the TOF depth data. 3D (D)={F i 3D ,i∈N}, that is, the feature vector of the tracer particle feature point corresponding to the TOF depth data in the target flow space is obtained. Where N is the total number of feature points in the TOF depth data; F i 3D is the spatial position of the i-th feature point in the TOF depth data, that is, the three-dimensional feature of the i-th feature point in the TOF depth data.

[0050] The three-dimensional feature extraction process of TOF depth data mainly considers the large reflectivity difference between the tracer particles and the flow field, and takes into account the change information of the edge of the tracer particles; in addition, the key point positions of the three-dimensional feature extraction of TOF depth data need to be stable to ensure that they can be repeatedly detected; using tracer particles as the detection area of ​​the three-dimensional features of TOF depth data ensures the stability of the corresponding descriptor and the uniqueness of the feature normal vector.

[0051] The specific method of extracting features from the infrared image data in step S2 is: extracting features from the infrared image data using a feature extraction operator to obtain a two-dimensional feature F of the infrared image data. 2D(I) = {R j ,j∈M}, that is, the feature vector of the tracer particle feature point corresponding to the infrared image data in the target flow space is obtained; the feature extraction operators include SIFT (Scale Invariant Feature Transform) operator, SURF (Speeded Up Robust Features) operator and ORB (Oriented Fast and Rotated Brief) operator; the two-dimensional features of the infrared image data include the image coordinates of the feature points and the feature descriptors of the feature points; M is the total number of feature points in the infrared image data; R j is the two-dimensional feature of the jth feature point in the infrared image data, R j ={s1,s2,s3,...,s K |(x j ,y j )},s1,s2,s3,...,s K is the feature descriptor corresponding to the feature point, (x j ,y j ) is the image coordinate corresponding to the feature point.

[0052] The specific method of step S3 is: normalize the two eigenvectors at the same time respectively, and form the two normalized eigenvectors at the same time into a complex vector F joint (D,I), and obtain the joint feature descriptor.

[0053] The specific method of step S4 includes the following sub-steps:

[0054] S4-1, taking the target feature point in the joint feature descriptor corresponding to time T0 as a reference, searching the feature point closest to the target feature point in the joint feature descriptor corresponding to time T1 through the KD tree to obtain a feature point pair; wherein time T0 and time T1 are adjacent sampling times;

[0055] S4-2, according to the formula:

[0056]

[0057] Get the joint feature similarity between time T0 and time T1 in is the feature descriptor of the cth feature point at time T0; is the feature descriptor of the cth feature point at time T1; N is the total number of feature points in the TOF depth data; M is the total number of feature points in the infrared image data;

[0058] S4-3, recording the feature point pairs whose joint feature similarity value is greater than the similarity threshold as feature points with the same name;

[0059] S4-4, through the random sampling consensus method, the feature points with the same name are divided into internal points and external points, and the corresponding relationship between the feature points with the same name with the smallest internal point error is taken as the time T0 and T1 moment The matching relationship between them.

[0060] The specific method of step S5 is: generate a three-dimensional motion vector and a three-dimensional velocity vector that change with time between feature points with the same name according to the matching relationship between feature points with the same name; use the three-dimensional motion vector to describe the coordinate change of the corresponding feature point in the three-dimensional space; use the three-dimensional velocity vector to describe the velocity of the corresponding feature point in the three-dimensional space, that is, obtain a sparse scene flow in the target flow space.

[0061] Step S6 also includes step S7: rendering, displaying and visualizing the interaction between the sparse scene flow and the dense scene flow; wherein the rendered sparse scene flow is used to track and locate the salient target in the target flow space; and the rendered dense scene flow is used to explain and understand the three-dimensional motion mechanism of the target flow space.

[0062] The visualization methods include three-dimensional arrow diagram, three-dimensional velocity diagram, flow tube diagram and flow belt diagram; and point cloud animations with different pseudo colors are used in the visualization process to represent the changes in the target flow space at different times.

[0063] In summary, the present invention uses a TOF device to directly shoot the target flow space and the tracing examples therein, and synchronously obtains TOF depth data and infrared image data at high speed, so as to obtain the scene flow of the entire three-dimensional fluid space in real time with high frame rate, high synchronization and high precision.

Claims

1. A scene flow acquisition method based on flight time, characterized in that: The following steps are involved: S1, obtaining TOF depth data and infrared image data at different times in the target flow space; S2, extracting features from the TOF depth data and the infrared image data respectively, and obtaining feature vectors of the feature points of the tracer particles in the target flow space; S3, combining the two feature vectors at the same time to obtain a joint feature descriptor; S4, through KD tree search and similarity measurement, feature matching is performed on the joint feature descriptors to obtain feature points with the same name and their matching relationships; S5, generating a three-dimensional motion vector and a three-dimensional velocity vector between the feature points with the same name that change over time based on the matching relationship between the feature points with the same name, that is, obtaining a sparse scene flow; S6, upsampling the sparse scene flow to obtain a dense scene flow; The specific method of step S4 includes the following sub-steps: S4-1, taking the target feature point in the joint feature descriptor corresponding to time T0 as a reference, searching the feature point closest to the target feature point in the joint feature descriptor corresponding to time T1 through the KD tree to obtain a feature point pair; wherein time T0 and time T1 are adjacent sampling times; S4-2, according to the formula: Get the joint feature similarity between time T0 and time T1 in is the feature descriptor of the cth feature point at time T0; is the feature descriptor of the cth feature point at time T1; N is the total number of feature points in the TOF depth data; M is the total number of feature points in the infrared image data; S4-3, recording the feature point pairs whose joint feature similarity value is greater than the similarity threshold as feature points with the same name; S4-4, through the random sampling consensus method, the feature points with the same name are divided into internal points and external points, and the corresponding relationship between the feature points with the same name with the smallest internal point error is taken as the time T0 and T1 moment The matching relationship between them.

2. The method for acquiring scene flow based on time of flight according to claim 1, characterized in that: The specific method of step S1 is: The TOF device emits a modulated light pulse signal in the infrared frequency band, and receives the modulated light pulse signal reflected by the target flow space and the tracer particles therein, thereby obtaining the TOF depth data and infrared image data at the corresponding moment.

3. The method for acquiring scene flow based on time of flight according to claim 1, characterized in that: The specific method for extracting features from TOF depth data in step S2 is: Based on the reflectivity difference between the tracer particles and the target flow space, a convolutional neural network is used to extract the spatial position of the tracer particles in the TOF depth data to obtain the three-dimensional features of the TOF depth data, that is, to obtain the feature vector of the tracer particle feature points corresponding to the TOF depth data in the target flow space.

4. The method for acquiring scene flow based on time of flight according to claim 1, characterized in that: The specific method for extracting features from infrared image data in step S2 is: Feature extraction operators are used to extract features from infrared image data to obtain two-dimensional features of the infrared image data, that is, feature vectors of tracer particle feature points corresponding to the infrared image data in the target flow space are obtained; the feature extraction operators include SIFT operators, SURF operators and ORB operators; the two-dimensional features of the infrared image data include the image coordinates of the feature points and the feature descriptors of the feature points.

5. The method for acquiring scene flow based on time of flight according to claim 1, characterized in that: The specific method of step S3 is: The two feature vectors at the same time are normalized respectively, and the two normalized feature vectors at the same time are combined into a complex vector to obtain a joint feature descriptor.

6. The method for acquiring scene flow based on time of flight according to claim 1, characterized in that: The specific method of step S5 is: According to the matching relationship of feature points with the same name, the three-dimensional motion vector and three-dimensional velocity vector that change with time between feature points with the same name are generated; the three-dimensional motion vector is used to describe the coordinate change of the corresponding feature point in the three-dimensional space; the three-dimensional velocity vector is used to describe the velocity of the corresponding feature point in the three-dimensional space, that is, the sparse scene flow in the target flow space is obtained.

7. The method for acquiring scene flow based on time of flight according to claim 1, characterized in that: Step S6 also includes step S7: rendering, displaying and visualizing the interaction between the sparse scene flow and the dense scene flow; wherein the rendered sparse scene flow is used to track and locate the salient target in the target flow space; The rendered dense scene flow is used to interpret and understand the target flow space.

8. The method for acquiring scene flow based on time of flight according to claim 7, characterized in that: Visualization methods include 3D arrow diagrams, 3D velocity diagrams, stream tube diagrams, and stream ribbon diagrams; In the visualization process, point cloud animations with different pseudo colors are used to represent the changes in the target flow space at different times.