Multipath marking method, multipath detection method, training method and their applications

Abstract

The present invention provides a multipath marking method, a multipath detection method, a training method and an application thereof, which can mark pixels affected by multipath interference with high precision without modifying hardware configuration. The multipath marking method comprises the steps of: obtaining depth measurement data of a TOF module in different scenes and depth true value data aligned to a TOF coordinate system; performing difference processing on the depth measurement data and the depth true value data to obtain depth difference data; judging whether the absolute pixel value of each pixel in the depth difference data is greater than a difference threshold, and pre-marking pixels whose absolute pixel values ​​are greater than the difference threshold as undetermined pixels affected by multipath interference; and re-screening the undetermined pixels based on optical path analysis to obtain multipath marking training data.

Description

Technical Field

[0001] The present invention relates to the technical field of TOF, and in particular to a multipath marking method, a multipath detection method, a training method and applications thereof. Background Art

[0002] TOF (Time Of Flight) technology is currently a common method for depth measurement. It has been widely used and popularized due to its high accuracy. With the development of robot applications, 3D imaging technology has been deeply integrated into applications such as sweepers and service robots. Among them, iTOF indirect time-of-flight imaging technology is one of the more mature solutions. iTOF emits infrared light, receives the light returned by the object, calculates the phase difference of the returned light, and indirectly calculates the flight time of the light to solve the distance of the object.

[0003] However, due to diffuse reflection of objects, the same measurement point will reflect light from different optical paths, resulting in the superposition of phase differences between light from different optical paths and the introduction of measurement errors caused by multipath. In particular, as the application scenarios of TOF depth calculation increase, the multipath interference problem of TOF itself has an increasingly greater impact on applications; if the pixels affected by multipath interference can be accurately identified and marked, it will be of great help to the application algorithm.

[0004] At present, the detection of pixels affected by multipath interference is mainly based on the difference in grayscale or depth of a single pixel at different modulation frequencies. This method not only has poor detection accuracy and a high false detection rate, but is also highly dependent on hardware configuration. Configuration parameters such as the choice of modulation frequency and duty cycle will affect the detection effect. Summary of the invention

[0005] An advantage of the present invention is to provide a multipath marking method, a multipath detection method, a training method and applications thereof, which can mark pixels affected by multipath interference with high precision without modifying hardware configuration.

[0006] Another advantage of the present invention is to provide a multipath marking method, a multipath detection method, a training method and their applications. In one embodiment of the present invention, the training method can realize the automation of the TOF multipath interference marking data set construction process and improve the efficiency of data set construction.

[0007] Another advantage of the present invention is that it provides a multipath marking method, a multipath detection method, a training method and applications thereof. In one embodiment of the present invention, the electronic device can use a deep learning algorithm to complete the marking of multipath interference effects, thereby achieving high-precision and real-time multipath interference detection, thereby reducing the requirements for platform performance.

[0008] Another advantage of the present invention is to provide a multipath marking method, a multipath detection method, a training method and their applications, wherein in order to achieve the above-mentioned purpose, no complicated structure and design are required in the present invention. Therefore, the present invention successfully and effectively provides a solution, not only providing a simple multipath marking method, a multipath detection method, a training method and their applications, but also increasing the practicality and reliability of the multipath marking method, the multipath detection method, the training method and their applications.

[0009] In order to achieve at least one of the above advantages or other advantages and purposes of the present invention, the present invention provides a multipath marking method, comprising the steps of:

[0010] Obtain the depth measurement data of the TOF module in different scenarios and the true depth data registered to the TOF coordinate system;

[0011] Performing a difference process on the depth measurement data and the depth true value data to obtain depth difference data;

[0012] Determine whether the absolute pixel value of each pixel in the depth difference data is greater than a difference threshold, and pre-mark pixels whose absolute pixel values ​​are greater than the difference threshold as pending pixels affected by multipath interference; and

[0013] Based on the optical path analysis, the undetermined pixel is screened again to obtain multipath label training data.

[0014] According to an embodiment of the present application, the step of re-screening the undetermined pixel based on optical path analysis to obtain multipath label training data includes the following steps:

[0015] A three-dimensional coordinate system is constructed with the optical center of the lens of the TOF module as the origin;

[0016] Fitting the plane equations of each sub-point cloud plane in the corresponding scene on the true depth data;

[0017] The three-dimensional coordinates of the undetermined pixel are used as the measurement light vector of the undetermined pixel, and the reflection light vector of the undetermined pixel is calculated in the corresponding scene; and

[0018] It is determined whether the reflected light path corresponding to the reflected light vector is a multipath interference light path, and the undetermined pixel corresponding to the multipath interference light path is finally marked as a determined pixel affected by multipath interference in the multipath marking training data.

[0019] According to an embodiment of the present application, the step of determining whether the reflected light path corresponding to the reflected light vector is a multipath interference light path, and finally marking the undetermined pixel corresponding to the multipath interference light path determined as a determined pixel affected by multipath interference in the multipath label training data, includes the steps of:

[0020] Determine whether the reflected light vector falls within the measurement light cone of the pixel to be determined;

[0021] In response to falling within the measurement light cone, determining that the undetermined pixel is affected by multipath interference, so as to mark the undetermined pixel as the determined pixel; and

[0022] In response to falling outside the measurement light cone, it is determined that the pending pixel is not affected by multipath interference, so as to cancel the mark of the pending pixel.

[0023] According to an embodiment of the present application, the step of determining whether the reflected light path corresponding to the reflected light vector is a multipath interference light path, and finally marking the undetermined pixel corresponding to the multipath interference light path determined as a determined pixel affected by multipath interference in the multipath label training data, includes the steps of:

[0024] Based on the plane equation in the scene corresponding to the undetermined pixel and the measured light vector, the amplitude, phase and distance of the direct path corresponding to the undetermined pixel are obtained;

[0025] Based on the plane equation in the scene corresponding to the undetermined pixel and the reflected light vector, the amplitude, phase and distance of the indirect path corresponding to the undetermined pixel are obtained;

[0026] Performing vector superposition of the direct path and the indirect path to obtain a multipath influence amount of the pixel to be determined;

[0027] Determine whether the multipath influence amount of the pixel to be determined is greater than the influence amount threshold;

[0028] In response to the multipath influence amount being greater than the influence amount threshold, determining that the undetermined pixel is interfered by multipath, so as to mark the undetermined pixel as the determined pixel; and

[0029] In response to the multipath influence amount being less than or equal to the influence amount threshold, it is determined that the pending pixel is not affected by multipath interference, so as to cancel the mark of the pending pixel.

[0030] According to an embodiment of the present application, the step of obtaining the amplitude, phase and distance of the indirect path corresponding to the undetermined pixel based on the plane equation in the scene corresponding to the undetermined pixel and the reflected light vector includes the steps of:

[0031] Calculate the intersection point between the reflected light vector and the plane equation as the reflection impact point to obtain the three-dimensional coordinates of the reflection impact point;

[0032] Calculate the distance between the reflection impact point and the center point of the plane corresponding to the pixel to be determined;

[0033] Extract the amplitude and distance of the reflection impact point;

[0034] Calculate the distance of the indirect path corresponding to the undetermined pixel through the indirect distance calculation model;

[0035] Calculating the phase of the indirect path corresponding to the undetermined pixel through the indirect phase calculation model; and

[0036] The amplitude of the indirect path corresponding to the pending pixel is calculated through the indirect amplitude calculation model.

[0037] According to one embodiment of the present application, the indirect distance calculation model is d F =(d0+d1+d3) / 2, where d F is the distance of the indirect path corresponding to the undetermined pixel; d0 is the straight-line distance between the undetermined pixel and the center point of the plane; d1 is the straight-line distance between the reflection influence point and the center point of the plane; d3 is the straight-line distance between the reflection influence point and the undetermined pixel.

[0038] According to one embodiment of the present application, the indirect phase calculation model is ψ F =(4πfd F ) / c, where ψ F is the phase of the indirect path corresponding to the pixel to be determined; f is the focal length of the lens of the TOF module; d F is the distance of the indirect path corresponding to the pixel to be determined; c is the speed of light.

[0039] According to one embodiment of the present application, the indirect amplitude calculation model is A F =(A Q d3 2 ) / d F 2 , where A F is the amplitude of the indirect path corresponding to the pixel to be determined; d3 is the straight-line distance between the reflection impact point and the pixel to be determined; d F is the distance of the indirect path corresponding to the pending pixel.

[0040] According to another aspect of the present application, the present application further provides a training method, comprising the steps of:

[0041] Constructing a training data set based on the multipath marking training data obtained by any of the multipath marking methods described above and the depth measurement data of the TOF module in different scenarios; and

[0042] The training data set is input into the labeling model system for network training to obtain a trained labeling model system.

[0043] According to another aspect of the present application, the present application further provides a multipath marking system, comprising:

[0044] The data acquisition module is used to obtain the depth measurement data of the TOF module in different scenes and the true depth data registered to the TOF coordinate system;

[0045] A depth difference module is used to perform a difference process on the depth measurement data and the depth true value data to obtain depth difference data;

[0046] A pre-marking judgment module is used to judge whether the absolute pixel value of each pixel in the depth difference data is greater than a difference threshold, and pre-mark the pixels whose absolute pixel values ​​are greater than the difference threshold as pending pixels affected by multipath interference; and

[0047] The screening module is used to screen the undetermined pixel again based on the optical path analysis to obtain multipath label training data.

[0048] According to one embodiment of the present application, the screening module includes: a coordinate system construction module, which is communicatively connected to each other, and is used to construct a three-dimensional coordinate system with the optical center of the lens of the TOF module as the origin; a plane fitting module, which is used to fit the plane equations of each sub-point cloud plane in the corresponding scene on the depth true value data; a vector calculation module, which is used to use the three-dimensional coordinates of the undetermined pixel as the measurement light vector of the undetermined pixel, and calculate the reflected light vector of the undetermined pixel in the corresponding scene; and a multipath judgment module, which is used to judge whether the reflected light path corresponding to the reflected light vector is a multipath interference light path, and finally mark the undetermined pixel corresponding to the multipath interference light path as a determined pixel affected by multipath interference in the multipath labeling training data.

[0049] According to one embodiment of the present application, the multipath judgment module includes: a cone judgment module that is communicatively connected to each other, used to judge whether the reflected light vector falls within the measurement light cone range of the pending pixel; a marking determination module, used to determine that the pending pixel is affected by multipath interference in response to falling within the measurement light cone range, so as to mark the pending pixel as the determined pixel; and a marking cancellation module, used to determine that the pending pixel is not affected by multipath interference in response to falling outside the measurement light cone range, so as to cancel the mark of the pending pixel.

[0050] According to one embodiment of the present application, the multipath determination module includes: a direct path acquisition module, which is communicatively connected to each other, and is used to obtain the amplitude, phase and distance of the direct path corresponding to the pixel to be determined based on the plane equation in the scene corresponding to the pixel to be determined and the measured light vector; an indirect path acquisition module, which is used to obtain the amplitude, phase and distance of the indirect path corresponding to the pixel to be determined based on the plane equation in the scene corresponding to the pixel to be determined and the reflected light vector; a vector superposition module, which is used to perform vector superposition of the direct path and the indirect path to obtain the multipath influence amount of the pixel to be determined; an influence amount determination module, which is used to determine whether the multipath influence amount of the pixel to be determined is greater than an influence amount threshold; a marking determination module, which is used to determine that the pixel to be determined is affected by multipath interference in response to the multipath influence amount being greater than the influence amount threshold, so as to mark the pixel to be determined as the determined pixel; and a marking cancellation module, which is used to determine that the pixel to be determined is not affected by multipath interference in response to the multipath influence amount being less than or equal to the influence amount threshold, so as to cancel the marking of the pixel to be determined.

[0051] According to one embodiment of the present application, the indirect path acquisition module includes: an influence point calculation module that is communicatively connected to each other, used to calculate the intersection between the reflected light vector and the plane equation as the reflection influence point to obtain the three-dimensional coordinates of the reflection influence point; a center distance calculation module, used to calculate the distance between the reflection influence point and the center point of the plane corresponding to the pixel to be determined; a reflection distance extraction module, used to extract the amplitude and distance of the reflection influence point; an indirect distance calculation module, used to calculate the distance of the indirect path corresponding to the pixel to be determined through an indirect distance calculation model; an indirect phase calculation module, used to calculate the phase of the indirect path corresponding to the pixel to be determined through an indirect phase calculation model; and an indirect amplitude calculation module, used to calculate the amplitude of the indirect path corresponding to the pixel to be determined through an indirect amplitude calculation model.

[0052] According to another aspect of the present application, the present application further provides a multipath detection method, comprising the steps of:

[0053] Collect depth measurement data through the TOF module; and

[0054] The depth measurement data is input into the label model system trained by the above training method to output a multipath label map.

[0055] According to another aspect of the present application, the present application further provides an electronic device, including:

[0056] TOF module; and

[0057] The marking model system trained by the above training method can be communicatively connected to the TOF module. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 is a flowchart of a multipath marking method according to an embodiment of the present application;

[0059] Figure 2 A schematic flow chart of the re-screening step in the multipath marking method according to the above embodiment of the present application is shown;

[0060] Figure 3 A first example of a multipath determination step in the multipath marking method according to the above embodiment of the present application is shown;

[0061] Figure 4 A schematic diagram showing the principle of the multipath determination step according to the first example of the present application is shown;

[0062] Figure 5 A second example of the multipath determination step in the multipath marking method according to the above embodiment of the present application is shown;

[0063] Figure 6 A schematic flow chart of an indirect path acquisition step in the multipath determination step according to the second example of the present application is shown;

[0064] Figure 7 A schematic diagram showing the principle of the indirect path acquisition step in the multipath determination step according to the second example of the present application is shown;

[0065] Figure 8 A schematic diagram showing the principle of the vector superposition step in the multipath determination step according to the second example of the present application is shown;

[0066] Fig. 9 is a flowchart of a training method according to an embodiment of the present application;

[0067] Fig.10 is a schematic block diagram of a multipath marking system according to an embodiment of the present application;

[0068] Fig.11 A first example of a multipath determination module in a multipath marking system according to the above embodiment of the present application is shown;

[0069] Fig.12 A second example of a multipath determination module in a multipath marking system according to the above embodiment of the present application is shown;

[0070] Fig.13 is a flowchart of a multipath detection method according to an embodiment of the present application;

[0071] Fig.14 It is a schematic block diagram of an electronic device according to an embodiment of the present application.

[0072] Explanation of main component symbols: 1. Multipath marking system; 10. Data acquisition module; 20. Depth difference module; 30. Pre-marking judgment module; 40. Screening module; 41. Coordinate system construction module; 42. Plane fitting module; 43. Vector calculation module; 44. Multipath judgment module; 441. Vision cone judgment module; 442. Marking determination module; 443. Marking cancellation module; 444. Direct path acquisition module; 445. Indirect path acquisition module; 4451. Impact point calculation module; 4452. Center distance calculation module; 4453. Reflection distance extraction module; 4454. Indirect distance calculation module; 4455. Indirect phase calculation module; 4456. Indirect amplitude calculation module; 446. Vector superposition module; 447. Impact quantity judgment module; 2. TOF module; 3. Marking model system.

[0073] The above description of the main component symbols is combined with the accompanying drawings and specific implementation methods to further illustrate the present invention in detail. DETAILED DESCRIPTION

[0074] The following description is used to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are only examples, and those skilled in the art can think of other obvious variations. The basic principles of the present invention defined in the following description can be applied to other embodiments, variations, improvements, equivalents, and other technical solutions that do not deviate from the spirit and scope of the present invention.

[0075] In the present invention, the term "one" in the claims and the specification should be understood as "one or more", that is, in one embodiment, the number of an element can be one, while in another embodiment, the number of the element can be multiple. Unless it is clearly indicated in the disclosure of the present invention that the number of the element is only one, the term "one" cannot be understood as unique or single, and the term "one" cannot be understood as a limitation on the quantity.

[0076] In the description of the present invention, it should be understood that "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, "connected" and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through a medium. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0077] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0078] Considering that the existing scheme judges by the difference of grayscale or depth of a single pixel at different modulation frequencies, not only the detection accuracy is poor and the false detection rate is high, but also it is particularly dependent on the hardware configuration, such as the selection of modulation frequency and configuration parameters such as duty cycle, which will affect the detection effect. In view of this, the present application creatively proposes a multipath marking method, a multipath detection method, a training method and its application, which can mark pixels affected by multipath interference with high precision without modifying the hardware configuration.

[0079] Specifically, refer to the accompanying drawings of the specification of this application. Figure 1 According to an embodiment of the present application, a multipath marking method is provided, which may include the steps of:

[0080] S100: Acquire depth measurement data of the TOF module in different scenes and register the depth true value data in the TOF coordinate system;

[0081] S200: performing a difference process on the depth measurement data and the depth true value data to obtain depth difference data;

[0082] S300: determining whether the absolute pixel value of each pixel in the depth difference data is greater than a difference threshold, and pre-marking pixels whose absolute pixel values ​​are greater than the difference threshold as pending pixels affected by multipath interference; and

[0083] S400: Based on the light path analysis, the undetermined pixel is screened again to obtain multipath label training data.

[0084] It is worth noting that although the undetermined pixels pre-marked simply by difference have the problems of poor accuracy and high false detection rate, this application re-screens the undetermined pixels through optical path analysis, and timely and efficiently removes the undetermined pixels that are not affected by multipath interference, which can not only improve the accuracy of multipath interference marking and reduce the false detection rate; but also has no high requirements for platform performance, and does not require modification of the existing hardware design, which can greatly improve the marking accuracy. It can be understood that the absolute pixel value mentioned in this application refers to the absolute value of each pixel value in the depth difference data; the TOF module mentioned in this application measures distance based on the iTOF principle.

[0085] In addition, the difference threshold mentioned in the present application can be determined according to the measurement distance and accuracy of the TOF module. For example, for a TOF module with a measurement distance of about 2m, its measurement accuracy is usually 1%, so the difference threshold of the present application can be implemented as 2cm but is not limited to.

[0086] More specifically, if Figure 2 As shown, step S400 of the present application may include the following steps:

[0087] S410: constructing a three-dimensional coordinate system with the optical center of the lens of the TOF module as the origin;

[0088] S420: Fitting the plane equations of each sub-point cloud plane in the corresponding scene on the true depth data;

[0089] S430: Using the three-dimensional coordinates of the undetermined pixel as the measurement light vector of the undetermined pixel, and calculating the reflection light vector of the undetermined pixel in the corresponding scene; and

[0090] S440: Determine whether the reflected light path corresponding to the reflected light vector is a multipath interference light path, and finally mark the undetermined pixel corresponding to the multipath interference light path as a determined pixel affected by multipath interference in the multipath label training data.

[0091] It is worth noting that since the measuring light will reflect according to the law of reflection when it meets the plane in the corresponding scene, the present application can calculate the corresponding reflected light vector based on the measuring light vector of the pending pixel and the plane equation of the corresponding plane; in this way, it is only necessary to determine whether the reflected light path corresponding to the reflected light vector is a multipath interference light path, so as to determine whether the corresponding pending pixel is affected by multipath interference, and then finally screen out the pending pixels that are truly affected by multipath interference as the determined pixels, and eliminate the pending pixels that are not affected by multipath interference, so as to improve the marking accuracy.

[0092] It can be understood that the optical center of the TOF module mentioned in this application refers to the origin of the mathematical model of the lens in the TOF module. The line connecting this point and the calibrated principal point of the TOF module is perpendicular to the image plane (i.e., the photosensitive surface of the photosensitive chip), and the distance from this point to the image plane is the focal length of the TOF module.

[0093] For example, in the first example of the present application, Figure 3 As shown, step S440 may include the following steps:

[0094] S441: Determine whether the reflected light vector falls within the measurement light cone range of the pixel to be determined;

[0095] S442: In response to falling within the measurement light cone, determining that the undetermined pixel is subject to multipath interference, and marking the undetermined pixel as the determined pixel; and

[0096] S443: In response to falling outside the measurement light cone, determining that the pending pixel is not affected by multipath interference, so as to cancel the mark of the pending pixel.

[0097] In other words, if the reflected light formed by the measurement light of the undetermined pixel after multiple reflections on the plane in the corresponding scene falls within the measurement light cone of the undetermined pixel, it can be determined that the undetermined pixel must be affected by multipath interference, so that the undetermined pixel is finally marked as a determined pixel affected by multipath interference in the multipath marking training data; and if the reflected light formed by the measurement light of the undetermined pixel after multiple reflections on the plane in the corresponding scene falls within the measurement light cone of the undetermined pixel, it can be determined that the undetermined pixel is not affected by multipath interference, and it is only necessary to cancel the mark of the undetermined pixel, ensuring that the pixels marked in the multipath marking training data are truly affected by multipath interference, which helps to improve the marking accuracy.

[0098] For example, taking a two-dimensional scene as an example: Figure 4 As shown in the figure, there are several pixel points on the image plane, and a pixel is represented by two adjacent vertical lines; for example, the pixel AB is between two points AB, and the fan-shaped area formed by OA and OB is the light receiving area of ​​the pixel AB. Light 1 falls within the light receiving area of ​​the pixel AB, and light 2 falls outside the light receiving range of the pixel AB. If the two-dimensional scene is expanded to a three-dimensional scene, the light receiving area of ​​the pixel AB is the measurement light cone range of the pixel AB.

[0099] It is worth mentioning that in the second example of this application, Figure 5 As shown, step S440 may also include the steps of:

[0100] S441′: based on the plane equation in the scene corresponding to the pixel to be determined and the measured light vector, obtaining the amplitude, phase and distance of the direct path corresponding to the pixel to be determined;

[0101] S442': based on other plane equations in the scene corresponding to the pixel to be determined and the reflected light vector, obtaining the amplitude, phase and distance of the indirect path corresponding to the pixel to be determined;

[0102] S443': performing vector superposition of the direct path and the indirect path to obtain the multipath influence amount of the pixel to be determined;

[0103] S444': Determine whether the multipath influence amount of the pixel to be determined is greater than the influence amount threshold;

[0104] S445′: In response to the multipath influence amount being greater than the influence amount threshold, determining that the undetermined pixel is interfered by multipath, and marking the undetermined pixel as the determined pixel; and

[0105] S446': In response to the multipath influence amount being less than or equal to the influence amount threshold, it is determined that the pending pixel is not affected by multipath interference, so as to cancel the mark of the pending pixel.

[0106] Optionally, in step S441': Figure 7 As shown, based on the three-dimensional coordinates of the pixel to be determined and the three-dimensional coordinates of the plane center point P corresponding to the pixel to be determined, the amplitude A of the direct path corresponding to the pixel to be determined is calculated. p , phase ψ P It can be understood that the distance d0 mentioned in this application refers to the straight-line distance between the pixel to be determined and the center point P of the plane; the amplitude A mentioned in this application p It refers to the total number of photons received by the pixel to be determined during the exposure time, which can be specifically calculated by The solution is: where I and Q represent the sum of the number of photons received at modulation phases of 0°+180° and 90°+270° respectively; the plane center point P mentioned in this application refers to the center point of the sub-point cloud plane corresponding to the pixel to be determined.

[0107] Alternatively, if Figure 6 and Figure 7 As shown, in step S442':

[0108] S4421′: Calculate the intersection point between the reflected light vector and the plane equation as the reflection influence point Q to obtain the three-dimensional coordinates of the reflection influence point Q;

[0109] S4422': Calculate the distance between the reflection influence point Q and the plane center point P corresponding to the pixel to be determined;

[0110] S4423': extract the amplitude and distance of the reflection impact point Q;

[0111] S4424′: Calculate the distance of the indirect path corresponding to the undetermined pixel through the indirect distance calculation model;

[0112] S4425': Calculate the phase of the indirect path corresponding to the undetermined pixel through the indirect phase calculation model; and S4426': Calculate the amplitude of the indirect path corresponding to the undetermined pixel through the indirect amplitude calculation model.

[0113] Optionally, the indirect distance calculation model may be implemented as, but not limited to, d F =(d0+d1+d3) / 2, where d F is the distance of the indirect path corresponding to the undetermined pixel; d0 is the straight-line distance between the undetermined pixel and the plane center point P corresponding to the undetermined pixel; d1 is the straight-line distance between the reflection influence point Q and the plane center point P; d3 is the straight-line distance between the reflection influence point Q and the undetermined pixel.

[0114] Optionally, the indirect phase calculation model can be implemented as, but not limited to, ψ F =(4πfd F ) / c, where ψ F is the phase of the indirect path corresponding to the pixel to be determined; f is the focal length of the TOF module lens; d F is the distance of the indirect path corresponding to the pixel to be determined; c is the speed of light.

[0115] Optionally, the indirect amplitude calculation model can be implemented as, but not limited to, A F =(A Q d3 2 ) / d F 2 , where A F is the amplitude of the indirect path corresponding to the pixel to be determined; d3 is the straight-line distance between the reflection influence point Q and the pixel to be determined; d F is the distance of the indirect path corresponding to the pending pixel.

[0116] It is worth noting that Figure 8 As shown, in step S443': the vector of the direct path is P(A P , ψ P ); the vector of the indirect path is F(A F , ψ F ); then the multipath influence vector P'(A P’ , ψ P’ ), and then calculate the multipath influence Δψ=ψ P’-ψ P .

[0117] In addition, in step S444': the influence threshold mentioned in the present application can be determined according to the phase ψ of the direct path corresponding to the pixel to be determined. P For example, the impact threshold of the present application may be, but is not limited to, implemented as 0.05*ψ P .

[0118] It can be understood that in the above-mentioned first example of the present application, the multipath marking method only performs qualitative marking, that is, it determines whether there is multipath reflected light returning to the pixel to be determined. If so, the pixel to be determined is marked as a determined pixel affected by multipath interference. If not, the mark of the pixel to be determined is cancelled; while in the above-mentioned second example of the present application, the multipath marking method performs quantitative marking, which can accurately mark whether the pixel to be determined is affected by multipath interference; especially for black materials, the multipath marking method in the above-mentioned second example can obtain a more accurate marking effect.

[0119] It is worth mentioning that according to another aspect of the present application, Fig. 9 As shown, an embodiment of the present application further provides a training method, which may include the steps of:

[0120] Constructing a training data set based on the multipath labeling training data obtained by the multipath labeling method and the depth measurement data of the TOF module in different scenarios; and

[0121] The training data set is input into the labeling model system for network training to obtain a trained labeling model system.

[0122] It is worth noting that the training data set mentioned in the present application preferably includes depth measurement data in various scenarios and multipath labeling training data in corresponding scenarios; the depth measurement data mentioned in the present application may be a depth map, an infrared map (IR map), RAW data and / or a point cloud map, etc., preferably implemented as a depth map and an IR map; the multipath labeling map mentioned in the present application may be implemented as a 0 / 1 labeling map, where 0 indicates that the corresponding pixel is not affected by multipath interference, and 1 indicates that the corresponding pixel is affected by multipath interference.

[0123] In this way, after the marking model system is trained according to the above method, the electronic device integrating the marking model system and the TOF module can mark the depth measurement data in real time through the marking model after collecting it through the TOF module, and obtain a multipath marking map, which helps to achieve high-precision and real-time multipath interference detection; compared with existing detection schemes, the detection scheme of the present application has high detection accuracy and low false detection rate, which can meet the requirements of actual use.

[0124] It is worth mentioning that according to another aspect of the present application, Fig.10 As shown, an embodiment of the present application further provides a multipath marking system 1, which may include: a data acquisition module 10, which is communicatively connected to each other, and is used to acquire depth measurement data of the TOF module in different scenes and depth true value data aligned to the TOF coordinate system; a depth difference module 20, which is used to perform difference processing on the depth measurement data and the depth true value data to obtain depth difference data; a pre-marking judgment module 30, which is used to judge whether the absolute pixel value of each pixel in the depth difference data is greater than a difference threshold, and pre-mark pixels whose absolute pixel values ​​are greater than the difference threshold as pending pixels affected by multipath interference; and a screening module 40, which is used to re-screen the pending pixels based on optical path analysis to obtain multipath marking training data.

[0125] Alternatively, if Fig.10 As shown, the screening module 40 includes: a coordinate system construction module 41, which is communicatively connected to each other, and is used to construct a three-dimensional coordinate system with the optical center of the lens of the TOF module as the origin; a plane fitting module 42, which is used to fit the plane equations of each sub-point cloud plane in the corresponding scene on the depth true value data; a vector calculation module 43, which is used to use the three-dimensional coordinates of the undetermined pixel as the measurement light vector of the undetermined pixel, and calculate the reflected light vector of the undetermined pixel in the corresponding scene; and a multipath judgment module 44, which is used to judge whether the reflected light path corresponding to the reflected light vector is a multipath interference light path, and finally mark the undetermined pixel corresponding to the multipath interference light path as a determined pixel affected by multipath interference in the multipath labeling training data.

[0126] For example, in the first example of the present application, Fig.11 As shown, the multipath judgment module 44 includes: a cone judgment module 441, which is communicatively connected to each other, and is used to judge whether the reflected light vector falls within the measurement light cone range of the pending pixel; a marking determination module 442, which is used to determine that the pending pixel is affected by multipath interference in response to falling within the measurement light cone range, so as to mark the pending pixel as the determined pixel; and a marking cancellation module 443, which is used to determine that the pending pixel is not affected by multipath interference in response to falling outside the measurement light cone range, so as to cancel the mark of the pending pixel.

[0127] In the second example of this application, Fig.12As shown, the multipath determination module 44 includes: a direct path acquisition module 444, which is communicatively connected to each other, and is used to obtain the amplitude, phase and distance of the direct path corresponding to the pixel to be determined based on the plane equation in the scene corresponding to the pixel to be determined and the measured light vector; an indirect path acquisition module 445, which is used to obtain the amplitude, phase and distance of the indirect path corresponding to the pixel to be determined based on the plane equation in the scene corresponding to the pixel to be determined and the reflected light vector; a vector superposition module 446, which is used to perform vector superposition of the direct path and the indirect path to obtain the multipath influence amount of the pixel to be determined; an influence amount determination module 447, which is used to determine whether the multipath influence amount of the pixel to be determined is greater than the influence amount threshold; a marking determination module 442, which is used to determine that the pixel to be determined is affected by multipath interference in response to the multipath influence amount being greater than the influence amount threshold, so as to mark the pixel to be determined as the determined pixel; and a marking cancellation module 443, which is used to determine that the pixel to be determined is not affected by multipath interference in response to the multipath influence amount being less than or equal to the influence amount threshold, so as to cancel the marking of the pixel to be determined.

[0128] Alternatively, if Fig.12 As shown, the indirect path acquisition module 445 includes: an influence point calculation module 4451, which is communicatively connected to each other, and is used to calculate the intersection between the reflected light vector and the plane equation as the reflection influence point to obtain the three-dimensional coordinates of the reflection influence point; a center distance calculation module 4452, which is used to calculate the distance between the reflection influence point and the center point of the plane corresponding to the pixel to be determined; a reflection distance extraction module 4453, which is used to extract the amplitude and distance of the reflection influence point; an indirect distance calculation module 4454, which is used to calculate the distance of the indirect path corresponding to the pixel to be determined through an indirect distance calculation model; an indirect phase calculation module 4455, which is used to calculate the phase of the indirect path corresponding to the pixel to be determined through an indirect phase calculation model; and an indirect amplitude calculation module 4456, which is used to calculate the amplitude of the indirect path corresponding to the pixel to be determined through an indirect amplitude calculation model.

[0129] It is worth mentioning that according to another aspect of the present application, Fig.13 As shown, an embodiment of the present application further provides a multipath detection method, comprising the steps of:

[0130] Collect depth measurement data through the TOF module; and

[0131] The depth measurement data is input into the label model system trained by the above training method to output a multipath label map.

[0132] It is worth mentioning that according to another aspect of the present application, Fig.14As shown, an embodiment of the present application further provides an electronic device, which may include a TOF module 2 and a marking model system 3 trained by the above-mentioned training method. The marking model system 3 can be communicatively connected to the TOF module 2 to mark the depth measurement data collected by the TOF module 2 in real time to obtain a multipath marking map, which helps to achieve high-precision and real-time multipath interference detection and can meet the requirements of actual use.

[0133] It is worth noting that the electronic device of the present application can be, but is not limited to, implemented as a TOF camera configured with the marking model system 3. It can also be implemented as equipment such as a sweeping robot, monitoring equipment, AR / VR, service robot, and personal verification equipment. As long as it is configured with the marking model system 3, this application will not go into details.

[0134] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0135] The above embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for those of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A multipath marking method, characterized in that: Includes steps: Obtain the depth measurement data of the TOF module in different scenarios and the true depth data registered to the TOF coordinate system; Performing a difference process on the depth measurement data and the depth true value data to obtain depth difference data; Determine whether the absolute pixel value of each pixel in the depth difference data is greater than a difference threshold, and pre-mark pixels whose absolute pixel values ​​are greater than the difference threshold as pending pixels affected by multipath interference; as well as Based on the optical path analysis, the undetermined pixel is screened again to obtain multipath label training data.

2. The multipath marking method according to claim 1, characterized in that: The step of re-screening the undetermined pixel based on the optical path analysis to obtain multipath label training data comprises the following steps: A three-dimensional coordinate system is constructed with the optical center of the lens of the TOF module as the origin; Fitting the plane equations of each sub-point cloud plane in the corresponding scene on the true depth data; The three-dimensional coordinates of the undetermined pixel are used as the measurement light vector of the undetermined pixel, and the reflection light vector of the undetermined pixel is calculated in the corresponding scene; as well as It is determined whether the reflected light path corresponding to the reflected light vector is a multipath interference light path, and the undetermined pixel corresponding to the multipath interference light path is finally marked as a determined pixel affected by multipath interference in the multipath marking training data.

3. The multipath marking method according to claim 2, characterized in that: The step of judging whether the reflected light path corresponding to the reflected light vector is a multipath interference light path, and finally marking the undetermined pixel corresponding to the multipath interference light path as a determined pixel affected by multipath interference in the multipath marking training data, comprises the steps of: Determine whether the reflected light vector falls within the measurement light cone of the pixel to be determined; In response to falling within the measurement light cone, determining that the undetermined pixel is affected by multipath interference, so as to mark the undetermined pixel as the determined pixel; as well as In response to falling outside the measurement light cone, it is determined that the pending pixel is not affected by multipath interference, so as to cancel the mark of the pending pixel.

4. The multipath marking method according to claim 2, characterized in that: The step of judging whether the reflected light path corresponding to the reflected light vector is a multipath interference light path, and finally marking the undetermined pixel corresponding to the multipath interference light path as a determined pixel affected by multipath interference in the multipath marking training data, comprises the steps of: Based on the plane equation in the scene corresponding to the undetermined pixel and the measured light vector, the amplitude, phase and distance of the direct path corresponding to the undetermined pixel are obtained; Based on the plane equation in the scene corresponding to the undetermined pixel and the reflected light vector, the amplitude, phase and distance of the indirect path corresponding to the undetermined pixel are obtained; Performing vector superposition of the direct path and the indirect path to obtain a multipath influence amount of the pixel to be determined; Determine whether the multipath influence amount of the pixel to be determined is greater than the influence amount threshold; In response to the multipath influence amount being greater than the influence amount threshold, determining that the undetermined pixel is interfered by multipath, and marking the undetermined pixel as the determined pixel; as well as In response to the multipath influence amount being less than or equal to the influence amount threshold, it is determined that the pending pixel is not affected by multipath interference, so as to cancel the mark of the pending pixel.

5. The multipath marking method according to claim 4, characterized in that: The step of obtaining the amplitude, phase and distance of the indirect path corresponding to the undetermined pixel based on the plane equation in the scene corresponding to the undetermined pixel and the reflected light vector comprises the steps of: Calculate the intersection point between the reflected light vector and the plane equation as the reflection impact point to obtain the three-dimensional coordinates of the reflection impact point; Calculate the distance between the reflection impact point and the center point of the plane corresponding to the pixel to be determined; Extract the amplitude and distance of the reflection impact point; Calculate the distance of the indirect path corresponding to the undetermined pixel through the indirect distance calculation model; Calculate the phase of the indirect path corresponding to the undetermined pixel through the indirect phase calculation model; as well as The amplitude of the indirect path corresponding to the pending pixel is calculated through the indirect amplitude calculation model.

6. The multipath marking method according to claim 5, characterized in that: The indirect distance calculation model is d F =(d0+d1+d3) / 2, where d F is the distance of the indirect path corresponding to the undetermined pixel; d0 is the straight-line distance between the undetermined pixel and the center point of the plane; d1 is the straight-line distance between the reflection influence point and the center point of the plane; d3 is the straight-line distance between the reflection influence point and the undetermined pixel.

7. The multipath marking method according to claim 5, characterized in that: The indirect phase calculation model is ψ F =(4πfd F ) / c, where ψ F is the phase of the indirect path corresponding to the pixel to be determined; f is the focal length of the lens of the TOF module; d F is the distance of the indirect path corresponding to the pixel to be determined; c is the speed of light.

8. The multipath marking method according to claim 5, characterized in that: The indirect amplitude calculation model is A F =(A Q d3 2 ) / d F 2 , where A F is the amplitude of the indirect path corresponding to the undetermined pixel; d3 is the straight-line distance between the reflection impact point and the undetermined pixel; d F is the distance of the indirect path corresponding to the pending pixel.

9. A training method, characterized in that Includes steps: Constructing a training data set based on the multipath labeling training data obtained by the multipath labeling method as described in any one of claims 1 to 8 and the depth measurement data of the TOF module in different scenarios; and The training data set is input into the labeling model system for network training to obtain a trained labeling model system.

10. A multipath marking system, characterized in that comprising communicatively connected to each other: The data acquisition module is used to obtain the depth measurement data of the TOF module in different scenes and the true depth data registered to the TOF coordinate system; A depth difference module is used to perform a difference process on the depth measurement data and the depth true value data to obtain depth difference data; A pre-marking judgment module is used to judge whether the absolute pixel value of each pixel in the depth difference data is greater than the difference threshold, and pre-mark the pixels whose absolute pixel values ​​are greater than the difference threshold as pending pixels affected by multipath interference; as well as The screening module is used to screen the undetermined pixel again based on the optical path analysis to obtain multipath label training data.

11. The multipath marking system according to claim 10, characterized in that: The screening module includes: a coordinate system construction module, which is communicatively connected to each other, and is used to construct a three-dimensional coordinate system with the optical center of the lens of the TOF module as the origin; a plane fitting module, which is used to fit the plane equations of each sub-point cloud plane in the corresponding scene on the depth true value data; A vector calculation module, used to use the three-dimensional coordinates of the undetermined pixel as the measurement light vector of the undetermined pixel, and calculate the reflection light vector of the undetermined pixel in the corresponding scene; And a multipath judgment module is used to judge whether the reflected light path corresponding to the reflected light vector is a multipath interference light path, and finally mark the undetermined pixel corresponding to the multipath interference light path as a determined pixel affected by multipath interference in the multipath label training data.

12. The multipath marking system according to claim 11, characterized in that: The multipath determination module includes: a cone determination module, which is communicatively connected to each other, for determining whether the reflected light vector falls within the measurement light cone range of the pixel to be determined; and a marking determination module, which is used to determine that the pixel to be determined is affected by multipath interference in response to falling within the measurement light cone range, so as to mark the pixel to be determined as the determined pixel; And a marking cancellation module is used for determining that the undetermined pixel is not affected by multipath in response to falling outside the measuring light cone range, so as to cancel the marking of the undetermined pixel.

13. The multipath marking system according to claim 11, characterized in that: The multipath determination module includes: a direct path acquisition module, which is communicatively connected to each other, and is used to acquire the amplitude, phase and distance of the direct path corresponding to the pixel to be determined based on the plane equation in the scene corresponding to the pixel to be determined and the measured light vector; an indirect path acquisition module, which is used to acquire the amplitude, phase and distance of the indirect path corresponding to the pixel to be determined based on the plane equation in the scene corresponding to the pixel to be determined and the reflected light vector; a vector superposition module, which is used to perform vector superposition of the direct path and the indirect path to obtain the multipath influence amount of the pixel to be determined; an influence amount determination module, which is used to determine whether the multipath influence amount of the pixel to be determined is greater than the influence amount threshold; a marking determination module, which is used to determine that the pixel to be determined is affected by multipath interference in response to the multipath influence amount being greater than the influence amount threshold, so as to mark the pixel to be determined as the determined pixel; And a marking cancellation module is used for determining that the pending pixel is not affected by multipath interference in response to the multipath influence amount being less than or equal to the influence amount threshold, so as to cancel the marking of the pending pixel.

14. The multipath marking system according to claim 13, characterized in that: The indirect path acquisition module includes: an influence point calculation module, which is communicatively connected to each other, and is used to calculate the intersection point between the reflection light vector and the plane equation as the reflection influence point to obtain the three-dimensional coordinates of the reflection influence point; a center distance calculation module, which is used to calculate the distance between the reflection influence point and the center point of the plane corresponding to the pixel to be determined; and a reflection distance extraction module, which is used to extract the amplitude and distance of the reflection influence point; An indirect distance calculation module, used to calculate the distance of the indirect path corresponding to the undetermined pixel through an indirect distance calculation model; An indirect phase calculation module, used to calculate the phase of the indirect path corresponding to the pixel to be determined through an indirect phase calculation model; And an indirect amplitude calculation module is used to calculate the amplitude of the indirect path corresponding to the undetermined pixel through an indirect amplitude calculation model.

15. A multipath detection method, characterized in that: Includes steps: Collect depth measurement data through the TOF module; and The depth measurement data is input into a label model system trained by the training method as claimed in claim 9 to output a multipath label map.

16. An electronic device, characterized in that include: TOF module; and The marking model system is trained by the training method as described in claim 9, and the marking model system is communicatively connected to the TOF module.

Images