Multipath detection method, multipath reduction method, system and electronic device thereof

Abstract

The present invention provides a multipath detection method, a multipath reduction method and a system and electronic device thereof, which can solve the problems of high platform performance requirements, large calculation amount, high hardware design complexity, etc. in the prior art. The multipath detection method includes the steps of: acquiring raw data collected by a TOF module, and calculating a point cloud map corresponding to the raw data; dividing the point cloud map into sub-point cloud planes according to the neighborhood window of each pixel to fit the plane equation of each sub-point cloud plane; obtaining the measurement light vector and the reflection light vector of the current pixel based on the three-dimensional coordinates of the current pixel and the center point and normal vector of the sub-point cloud plane corresponding to the current pixel; calculating the multipath influence amount of the current pixel based on the measurement light vector and the reflection light vector; and judging whether the multipath influence amount is greater than the influence amount threshold, if so, determining that the current pixel is an abnormal pixel, if not, determining that the current pixel is a normal pixel.

Description

Technical Field

[0001] The present invention relates to the field of TOF technology, and in particular to a multipath detection method, a multipath reduction method, a system thereof and an electronic device. Background Art

[0002] With the development of robot applications, 3D imaging technology has been deeply applied to sweepers, service robots and other applications, among which iTOF indirect time-of-flight imaging technology is one of the more mature solutions. As we all know, iTOF calculates the distance of an object by emitting infrared light, receiving the light returned by the object, calculating the phase difference of the returned light, and indirectly calculating the flight time of the light.

[0003] However, due to the diffuse reflection of objects, the same measurement point will reflect light from different optical paths, resulting in the superposition of phase differences between light from different optical paths and the introduction of measurement errors caused by multipaths. Therefore, iTOF is susceptible to multipath interference, which will lead to a decrease in measurement accuracy in practical applications such as robot obstacle avoidance and mapping.

[0004] At present, the technical solution to eliminate multipath interference is mainly to correct the measured depth affected by multipath interference to eliminate multipath interference. Specific methods usually include deep learning, multi-frequency modulation settlement or adding structured light sources. Among them, the deep learning method requires a large amount of learning data and true values ​​to train the network model, which requires a lot of resources and has high requirements on platform performance when deployed on the platform side. It usually requires NPU and CPU support, has high dependence on the platform, and has high application costs; the multi-frequency modulation solution method is to extract direct components through algorithms such as signal sparse characteristics and compressed sensing to remove the impact of multipath and obtain correct depth information, but this has problems such as large amount of calculation and difficulty in platform landing; the method of adding structured light sources is to reduce the impact of multipath by inducing triangulation and other methods, but there are problems such as high design complexity, high hardware cost, and sparse depth generation. Summary of the invention

[0005] One advantage of the present invention is that it provides a multipath detection method, a multipath reduction method, a system thereof, and an electronic device, which can solve the problems of the prior art in terms of high platform performance requirements, large amount of calculation, and high hardware design complexity.

[0006] Another advantage of the present invention is to provide a multipath detection method, a multipath reduction method, a system and an electronic device thereof, wherein, in one embodiment of the present invention, the multipath reduction method can eliminate point cloud data with large errors so as to solve the problem of poor obstacle avoidance and mapping accuracy caused by the influence of multipath.

[0007] Another advantage of the present invention is that it provides a multipath detection method, a multipath mitigation method, a system thereof, and an electronic device. In one embodiment of the present invention, the multipath mitigation method can calculate the degree to which the current pixel is affected by multipath, so as to filter out pixels with more serious impact, avoid large errors caused by multipath interference, and significantly reduce the impact of multipath interference.

[0008] Another advantage of the present invention is that it provides a multipath detection method, a multipath mitigation method, a system and an electronic device thereof. In one embodiment of the present invention, the multipath interference detection method can only use the point cloud data originally generated by the TOF module without the need to add multiple sets of frequencies and other hardware modifications, and has no high requirements on platform performance.

[0009] Another advantage of the present invention is to provide a multipath detection method, a multipath mitigation method, a system thereof, and an electronic device, wherein in order to achieve the above-mentioned purpose, no complicated structure and design are required in the present invention. Therefore, the present invention successfully and effectively provides a solution, not only providing a simple multipath detection method, a multipath mitigation method, a system thereof, and an electronic device, but also increasing the practicality and reliability of the multipath detection method, the multipath mitigation method, the system thereof, and the electronic device.

[0010] In order to achieve at least one of the above advantages or other advantages and purposes of the present invention, the present invention provides a multipath detection method, comprising the steps of:

[0011] Obtaining the original data collected by the TOF module and calculating the point cloud map corresponding to the original data;

[0012] The point cloud image is divided into sub-point cloud planes according to the neighborhood window of each pixel, so as to fit the plane equation of each sub-point cloud plane;

[0013] Based on the three-dimensional coordinates of the current pixel and the center point and normal vector of the sub-point cloud plane corresponding to the current pixel, a measurement light vector and a reflection light vector of the current pixel are obtained;

[0014] Calculating the multipath influence amount of the current pixel based on the measured light vector and the reflected light vector; and

[0015] It is determined whether the multipath influence amount is greater than an influence amount threshold. If so, the current pixel is determined to be an abnormal pixel; if not, the current pixel is determined to be a normal pixel.

[0016] According to one embodiment of the present application, the step of obtaining the measurement light vector and the reflection light vector of the current pixel based on the three-dimensional coordinates of the current pixel and the center point and the normal vector of the sub-point cloud plane corresponding to the current pixel includes the steps of:

[0017] A three-dimensional coordinate system is constructed with the optical center of the lens of the TOF module as the origin;

[0018] Extracting the three-dimensional coordinates of the current pixel as a measurement light vector of the current pixel;

[0019] Extracting the three-dimensional coordinates of the center point of the sub-point cloud plane corresponding to the current pixel and the normal vector of the sub-point cloud plane to construct an incident plane; and

[0020] Based on the measured light vector and the incident plane, the reflected light vector of the current pixel is calculated.

[0021] According to an embodiment of the present application, the step of calculating the multipath influence amount of the current pixel based on the measured light vector and the reflected light vector includes the steps of:

[0022] Based on the plane equation of the sub-point cloud plane corresponding to the current pixel and the measurement light vector, acquiring the amplitude, phase and distance of the direct path corresponding to the current pixel;

[0023] Based on the plane equations of the reflected light vector and other sub-point cloud planes, obtaining the amplitude, phase and distance of the indirect path corresponding to the current pixel; and

[0024] The direct path and the indirect path are vector-superimposed to obtain the multipath influence amount of the current pixel.

[0025] According to an embodiment of the present application, the step of obtaining the amplitude, phase and distance of the indirect path corresponding to the current pixel based on the plane equations of the reflected light vector and other sub-point cloud planes includes the steps of:

[0026] Calculate the intersection point between the reflected light vector and other plane equations as the reflection impact point to obtain the three-dimensional coordinates of the reflection impact point;

[0027] Calculate the distance between the reflection impact point and the center point of the sub-point cloud plane corresponding to the current pixel;

[0028] Extract the amplitude and distance of the reflection impact point;

[0029] Calculate the distance of the indirect path corresponding to the current pixel through the indirect distance calculation model;

[0030] Calculating the phase of the indirect path corresponding to the current pixel through the indirect phase calculation model; and

[0031] The amplitude of the indirect path corresponding to the current pixel is calculated through the indirect amplitude calculation model.

[0032] According to one embodiment of the present application, the indirect distance calculation model is dF=(d0+d1+d3) / 2, where dF is the distance of the indirect path corresponding to the current pixel; d0 is the straight-line distance between the current pixel and the center point of the sub-point cloud plane corresponding to the current pixel; d1 is the straight-line distance between the reflection influence point and the center point; d3 is the straight-line distance between the reflection influence point and the current pixel.

[0033] According to one embodiment of the present application, the indirect phase calculation model is ψF=(4πfdF) / c, where ψF is the phase of the indirect path corresponding to the current pixel; f is the lens focal length of the TOF module; dF is the distance of the indirect path corresponding to the current pixel; and c is the speed of light.

[0034] According to one embodiment of the present application, the indirect amplitude calculation model is AF=(AQd32) / dF2, where AF is the amplitude of the indirect path corresponding to the current pixel; d3 is the straight-line distance between the reflection impact point and the current pixel; and dF is the distance of the indirect path corresponding to the current pixel.

[0035] According to another aspect of the present application, the present application further provides a multipath mitigation method, comprising the steps of:

[0036] Detecting abnormal pixels and normal pixels in the raw data collected by the TOF module by any of the multipath detection methods described above; and

[0037] The abnormal pixels are filtered out to retain the normal pixels and obtain multipath interference reduction data.

[0038] According to another aspect of the present application, the present application further provides a multipath detection system, comprising:

[0039] A data acquisition module is used to acquire the original data collected by the TOF module and calculate a point cloud map corresponding to the original data;

[0040] A plane fitting module is used to divide the point cloud image into sub-point cloud planes according to the neighborhood window of each pixel, so as to fit the plane equation of each sub-point cloud plane;

[0041] A vector acquisition module, used for obtaining a measurement light vector and a reflection light vector of the current pixel based on the three-dimensional coordinates of the current pixel and the center point and normal vector of the sub-point cloud plane corresponding to the current pixel;

[0042] an influence amount calculation module, used to calculate the multipath influence amount of the current pixel based on the measured light vector and the reflected light vector; and

[0043] The threshold judgment module is used to judge whether the multipath influence amount is greater than the influence amount threshold. If so, the current pixel is determined to be an abnormal pixel; if not, the current pixel is determined to be a normal pixel.

[0044] According to one embodiment of the present application, the vector acquisition module includes: a coordinate system construction module that is communicatively connected to each other, used to construct a three-dimensional coordinate system with the lens optical center of the TOF module as the origin; a measurement vector extraction module, used to extract the three-dimensional coordinates of the current pixel as the measurement light vector of the current pixel; an incident plane construction module, used to extract the three-dimensional coordinates of the center point of the sub-point cloud plane corresponding to the current pixel and the normal vector of the sub-point cloud plane to construct the incident plane; and a reflection vector calculation module, used to calculate the reflection light vector of the current pixel based on the measurement light vector and the incident plane.

[0045] According to one embodiment of the present application, the influence amount calculation module includes: a direct path acquisition module, which is communicatively connected to each other, and is used to obtain the amplitude, phase and distance of the direct path corresponding to the current pixel based on the plane equation of the sub-point cloud plane corresponding to the current pixel and the measured light vector; an indirect path acquisition module, which is used to obtain the amplitude, phase and distance of the indirect path corresponding to the current pixel based on the reflected light vector and the plane equations of other sub-point cloud planes; and a vector superposition module, which is used to perform vector superposition of the direct path and the indirect path to obtain the multipath influence amount of the current pixel.

[0046] According to one embodiment of the present application, the indirect path acquisition module includes: an influence point calculation module that is communicatively connected to each other, used to calculate the intersection between the reflected light vector and other plane equations as the reflection influence point to obtain the three-dimensional coordinates of the reflection influence point; a center distance calculation module, used to calculate the distance between the reflection influence point and the center point of the sub-point cloud plane corresponding to the current pixel; a reflection distance extraction module, used to extract the amplitude and distance of the reflection influence point; an indirect distance calculation module, used to calculate the distance of the indirect path corresponding to the current pixel through an indirect distance calculation model; an indirect phase calculation module, used to calculate the phase of the indirect path corresponding to the current pixel through an indirect phase calculation model; and an indirect amplitude calculation module, which calculates the amplitude of the indirect path corresponding to the current pixel through an indirect amplitude calculation model.

[0047] According to another aspect of the present application, the present application further provides a multipath mitigation system, comprising:

[0048] Any of the multipath detection systems described above, used to detect abnormal pixels and normal pixels in the raw data collected by the TOF module; and

[0049] The filtering module is used to filter out the abnormal pixels to retain the normal pixels and obtain multipath interference reduction data.

[0050] According to another aspect of the present application, the present application further provides an electronic device, including:

[0051] TOF module; and

[0052] The multipath mitigation system mentioned above can be communicatively connected to the TOF module. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 is a flowchart of a multipath detection method according to an embodiment of the present application;

[0054] Figure 2 A schematic flow chart of a vector acquisition step in a multipath detection method according to the above embodiment of the present application is shown;

[0055] Figure 3 A schematic flow chart of the influence amount calculation step in the multipath detection method according to the above embodiment of the present application is shown;

[0056] Figure 4 An example of an interval path acquisition step in the influence amount calculation step according to the above embodiment of the present application is shown;

[0057] Figure 5 A schematic diagram showing the principle of the indirect path acquisition step in the multipath detection method according to the above embodiment of the present application;

[0058] Figure 6 A schematic diagram showing the principle of the vector superposition step in the multipath detection method according to the above embodiment of the present application is shown;

[0059] Figure 7 is a flowchart of a multipath mitigation method according to an embodiment of the present application;

[0060] Figure 8 is a schematic block diagram of a multipath detection system according to an embodiment of the present application;

[0061] Fig. 9 is a schematic block diagram of a multipath mitigation system according to an embodiment of the present application;

[0062] Fig.10 It is a schematic block diagram of an electronic device according to an embodiment of the present application.

[0063] Explanation of main component symbols: 1. Multipath detection system; 10. Data acquisition module; 20. Plane fitting module; 30. Vector acquisition module; 31. Coordinate system construction module; 32. Measurement vector extraction module; 33. Incident plane construction module; 34. Reflection vector calculation module; 40. Influence quantity calculation module; 41. Direct path acquisition module; 42. Indirect path acquisition module; 421. Influence point calculation module; 422. Center distance calculation module; 423. Reflection distance extraction module; 424. Indirect distance calculation module; 425. Indirect phase calculation module; 426. Indirect amplitude calculation module; 43. Vector superposition module; 50. Threshold judgment module; 2. TOF module; 3. Filtering module.

[0064] The above description of the main component symbols is combined with the accompanying drawings and specific implementation methods to further illustrate the present invention in detail. DETAILED DESCRIPTION

[0065] The following description is used to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are only examples, and those skilled in the art can think of other obvious variations. The basic principles of the present invention defined in the following description can be applied to other embodiments, variations, improvements, equivalents, and other technical solutions that do not deviate from the spirit and scope of the present invention.

[0066] In the present invention, the term "one" in the claims and the specification should be understood as "one or more", that is, in one embodiment, the number of an element can be one, while in another embodiment, the number of the element can be multiple. Unless it is clearly indicated in the disclosure of the present invention that the number of the element is only one, the term "one" cannot be understood as unique or single, and the term "one" cannot be understood as a limitation on the quantity.

[0067] In the description of the present invention, it should be understood that "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, "connected" and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through a medium. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0068] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0069] Considering the existing solutions' high requirements on platform performance, large amount of calculation, high hardware design complexity, etc. In order to solve this problem, the present application creatively proposes a multipath detection method, a multipath reduction method, a system and an electronic device thereof.

[0070] Specifically, refer to the accompanying drawings of the specification of this application. Figure 1 According to an embodiment of the present application, a multipath detection method is provided, which may include the steps of:

[0071] S100: Acquire raw data collected by the TOF module, and calculate a point cloud map corresponding to the raw data;

[0072] S200: Dividing the point cloud image into sub-point cloud planes according to the neighborhood window of each pixel, so as to fit the plane equation of each sub-point cloud plane;

[0073] S300: obtaining a measurement light vector and a reflection light vector of the current pixel based on the three-dimensional coordinates of the current pixel and the center point and normal vector of the sub-point cloud plane corresponding to the current pixel;

[0074] S400: Calculating the multipath influence amount of the current pixel based on the measured light vector and the reflected light vector; and

[0075] S500: Determine whether the multipath influence is greater than an influence threshold. If so, determine that the current pixel is an abnormal pixel. If not, determine that the current pixel is a normal pixel.

[0076] More specifically, the influence threshold mentioned in the present application can be calculated based on the phase ψ of the direct path corresponding to the current pixel. P For example, the impact threshold of the present application may be, but is not limited to, implemented as 0.05*ψ P It can be understood that the phase ψ of the direct path mentioned in this application is P It can be calculated from the measured light vector of the current pixel.

[0077] In this embodiment of the present application, the abnormal pixel refers to a pixel whose multipath influence amount is greater than the influence amount threshold, that is, a pixel that is greatly affected by multipath interference (a pixel that is truly affected by multipath interference); the normal pixel refers to a pixel whose multipath influence amount is less than or equal to the influence amount threshold, that is, a pixel that is not affected by multipath interference or is less affected by multipath interference (a pixel that is not truly affected by multipath interference).

[0078] Optionally, in step S100 of the present application, an amplitude map, a phase map and a depth map may also be calculated based on the original data.

[0079] It is worth noting that the multipath detection method of the present application can utilize a simplified optical path analysis method, and only needs to use the point cloud data originally generated by the TOF module to quantitatively detect the multipath interference pixels, so as to subsequently filter out and eliminate the point cloud data with large errors, thereby effectively solving the problem of poor obstacle avoidance and mapping accuracy in applications such as robots due to the influence of multipath in real applications; there is no need to add multiple sets of frequencies or other hardware modifications, which is conducive to reducing hardware costs. It is understandable that the TOF module mentioned in this application measures distance through the iTOF principle; the original data mentioned in this application refers to RAW data.

[0080] In addition, the neighborhood window of each pixel mentioned in the present application refers to an n*n neighborhood window centered on the current pixel, where n can be implemented as a positive integer greater than or equal to 2; for example, a 3*3 neighborhood window.

[0081] For example, Figure 2 As shown, step S300 of the present application may include the following steps:

[0082] S310: constructing a three-dimensional coordinate system with the optical center of the lens of the TOF module as the origin;

[0083] S320: extracting the three-dimensional coordinates of the current pixel as the measurement light vector of the current pixel;

[0084] S330: extracting the three-dimensional coordinates of the center point P of the sub-point cloud plane corresponding to the current pixel and the normal vector of the sub-point cloud plane to construct an incident plane; and

[0085] S340: Calculate the reflected light vector of the current pixel based on the measured light vector and the incident plane.

[0086] It is worth noting that since the measuring light will reflect according to the law of reflection when it meets the sub-point cloud plane in the point cloud image, the present application can calculate the corresponding reflected light vector based on the measuring light vector of the current pixel and the corresponding incident plane; in this way, it is only necessary to determine whether the reflected light path corresponding to the reflected light vector is a multipath interference light path, so as to determine whether the corresponding current pixel is affected by multipath interference, and finally screen out the pixels that are truly affected by multipath interference so as to filter them out later and eliminate the influence of multipath interference.

[0087] It can be understood that the optical center of the TOF module lens mentioned in this application refers to the origin of the mathematical model of the lens in the TOF module. The line connecting this point and the calibrated principal point of the TOF module is perpendicular to the image plane (i.e., the photosensitive surface of the photosensitive chip), and the distance from this point to the image plane is the focal length of the TOF module.

[0088] In one example of this application, Figure 3 As shown, step S400 of the present application may include the following steps:

[0089] S410: Acquire the amplitude, phase and distance of the direct path corresponding to the current pixel based on the plane equation of the sub-point cloud plane corresponding to the current pixel and the measurement light vector;

[0090] S420: Based on the plane equations of the reflected light vector and other sub-point cloud planes, obtain the amplitude, phase and distance of the indirect path corresponding to the current pixel; and

[0091] S430: Perform vector superposition of the direct path and the indirect path to obtain a multipath influence amount of the current pixel.

[0092] Optionally, in step S410: Figure 5 As shown, based on the three-dimensional coordinates of the current pixel and the three-dimensional coordinates of the center point P of the sub-point cloud plane corresponding to the current pixel, the amplitude A of the direct path corresponding to the current pixel is calculated. p , phase ψ P It can be understood that the distance d0 mentioned in this application refers to the straight-line distance between the current pixel and the center point P; the amplitude A mentioned in this application p Refers to the total number of photons received by the current pixel during the exposure time, which can be specifically calculated by The solution is: where I and Q represent the sum of the number of photons received at modulation phases of 0°+180° and 90°+270°, respectively.

[0093] Alternatively, if Figure 4 and Figure 5 As shown, in step S420:

[0094] S421: Calculate the intersection point between the reflected light vector and other plane equations as the reflection influence point Q to obtain the three-dimensional coordinates of the reflection influence point Q;

[0095] S422: Calculate the distance between the reflection influence point Q and the center point P of the sub-point cloud plane corresponding to the current pixel;

[0096] S423: extracting the amplitude and distance of the reflection impact point Q;

[0097] S424: Calculate the distance of the indirect path corresponding to the current pixel through the indirect distance calculation model;

[0098] S425: Calculate the phase of the indirect path corresponding to the current pixel through the indirect phase calculation model; and

[0099] S426: Calculate the amplitude of the indirect path corresponding to the current pixel through the indirect amplitude calculation model.

[0100] Optionally, the indirect distance calculation model may be implemented as, but not limited to, d F =(d0+d1+d3) / 2, where d F is the distance of the indirect path corresponding to the current pixel; d0 is the straight-line distance between the current pixel and the center point P of the sub-point cloud plane corresponding to the current pixel; d1 is the straight-line distance between the reflection influence point Q and the center point P; d3 is the straight-line distance between the reflection influence point Q and the current pixel.

[0101] Optionally, the indirect phase calculation model can be implemented as, but not limited to, ψ F =(4πfd F ) / c, where ψ F is the phase of the indirect path corresponding to the current pixel; f is the focal length of the TOF module lens; d F is the distance of the indirect path corresponding to the current pixel; c is the speed of light.

[0102] Optionally, the indirect amplitude calculation model can be implemented as, but not limited to, A F =(A Q d3 2 ) / d F 2 , where A F is the amplitude of the indirect path corresponding to the current pixel; d3 is the straight-line distance between the reflection influence point Q and the current pixel; d F is the distance of the indirect path corresponding to the current pixel.

[0103] It is worth noting that Figure 6As shown, in step S430: the vector of the direct path is P(A P , ψ P ); the vector of the indirect path is F(A F , ψ F ); then the multipath influence vector P'(A P’ , ψ P’ ), and then calculate the multipath influence Δψ=ψ P’ -ψ P .

[0104] It is worth mentioning that according to another aspect of the present application, Figure 7 As shown, an embodiment of the present application further provides a multipath mitigation method, which may include the steps of:

[0105] By using the multipath detection method, abnormal pixels and normal pixels in the raw data collected by the TOF module are detected; and

[0106] The abnormal pixels are filtered out to retain the normal pixels and obtain multipath interference reduction data.

[0107] In other words, in response to the multipath influence amount being greater than the influence amount threshold, it is determined that the current pixel is greatly affected by multipath interference and is regarded as an abnormal pixel, and after removing the abnormal pixel, the next pixel is detected for multipath interference; in response to the multipath influence amount being less than or equal to the influence amount threshold, it is determined that the current pixel is not affected by multipath interference or is less affected by multipath interference and is regarded as a normal pixel, and after retaining the normal pixel, the next pixel is detected for multipath interference.

[0108] Optionally, in this embodiment of the present application, the multipath mitigation method of the present application may be, but is not limited to, filtering out ground convolution using multipath influence amounts, so as to better filter out pixels affected by thin multipath interference (ie, the abnormal pixels).

[0109] In summary, the multipath detection method of the present application quantitatively detects the impact of multipath interference and can accurately detect whether the current pixel is greatly affected by multipath interference; especially for conventional scenes with ordinary reflectivity such as black materials, the above multipath detection method can obtain a more accurate detection effect, so that the multipath reduction method can accurately filter out abnormal pixels, thereby significantly reducing the impact of multipath interference.

[0110] It is worth mentioning that according to another aspect of the present application, Figure 8As shown, an embodiment of the present application further provides a multipath detection system 1, which may include: a data acquisition module 10, which is communicatively connected to each other, and is used to acquire the original data collected by the TOF module, and calculate a point cloud map corresponding to the original data; a plane fitting module 20, which is used to divide the point cloud map into various sub-point cloud planes according to the neighborhood window of each pixel, so as to fit the plane equations of each sub-point cloud plane; a vector acquisition module 30, which is used to obtain the measurement light vector and the reflection light vector of the current pixel based on the three-dimensional coordinates of the current pixel and the center point and normal vector of the sub-point cloud plane corresponding to the current pixel; an influence amount calculation module 40, which is used to calculate the multipath influence amount of the current pixel based on the measurement light vector and the reflection light vector; and a threshold judgment module 50, which is used to judge whether the multipath influence amount is greater than the influence amount threshold, and if so, determine that the current pixel is an abnormal pixel, and if not, determine that the current pixel is a normal pixel.

[0111] In one example of this application, Figure 8 As shown, the vector acquisition module 30 may include: a coordinate system construction module 31 that is communicatively connected to each other, for constructing a three-dimensional coordinate system with the optical center of the lens of the TOF module as the origin; a measurement vector extraction module 32, for extracting the three-dimensional coordinates of the current pixel as the measurement light vector of the current pixel; an incident plane construction module 33, for extracting the three-dimensional coordinates of the center point of the sub-point cloud plane corresponding to the current pixel and the normal vector of the sub-point cloud plane to construct the incident plane; and a reflection vector calculation module 34, for calculating the reflection light vector of the current pixel based on the measurement light vector and the incident plane.

[0112] In one example of this application, Figure 8 As shown, the influence amount calculation module 40 may include: a direct path acquisition module 41, which is communicatively connected to each other, and is used to obtain the amplitude, phase and distance of the direct path corresponding to the current pixel based on the plane equation of the sub-point cloud plane corresponding to the current pixel and the measured light vector; an indirect path acquisition module 42, which is used to obtain the amplitude, phase and distance of the indirect path corresponding to the current pixel based on the reflected light vector and the plane equations of other sub-point cloud planes; and a vector superposition module 43, which is used to perform vector superposition of the direct path and the indirect path to obtain the multipath influence amount of the current pixel.

[0113] Alternatively, if Figure 8As shown, the indirect path acquisition module 42 includes: an influence point calculation module 421, which is communicatively connected to each other, and is used to calculate the intersection between the reflected light vector and other plane equations as the reflection influence point to obtain the three-dimensional coordinates of the reflection influence point; a center distance calculation module 422, which is used to calculate the distance between the reflection influence point and the center point of the sub-point cloud plane corresponding to the current pixel; a reflection distance extraction module 423, which is used to extract the amplitude and distance of the reflection influence point; an indirect distance calculation module 424, which is used to calculate the distance of the indirect path corresponding to the current pixel through an indirect distance calculation model; an indirect phase calculation module 425, which is used to calculate the phase of the indirect path corresponding to the current pixel through an indirect phase calculation model; and an indirect amplitude calculation module 426, which calculates the amplitude of the indirect path corresponding to the current pixel through an indirect amplitude calculation model.

[0114] It is worth mentioning that according to another aspect of the present application, Fig. 9 As shown, an embodiment of the present application further provides a multipath mitigation system, which may include the above-mentioned multipath detection system 1 and filtering module 3 that are communicatively connected to each other; the multipath detection system 1 is used to detect abnormal pixels and normal pixels in the original data collected by the TOF module; the filtering module 3 is used to filter out the abnormal pixels to retain the normal pixels and obtain multipath interference mitigation data.

[0115] It is worth mentioning that according to another aspect of the present application, Fig.10 As shown, an embodiment of the present application further provides an electronic device, which may include a TOF module 2 and the above-mentioned multipath mitigation system. The multipath mitigation system can be communicatively connected to the TOF module 2 to detect the original data collected by the TOF module 2 in real time, and filter out pixels that are greatly affected by multipath interference (i.e., abnormal pixels), which helps to achieve high-precision, real-time multipath interference mitigation and can meet the requirements of actual use.

[0116] It is worth noting that the electronic device of the present application can be, but is not limited to, implemented as a TOF camera equipped with the multipath mitigation system. It can also be implemented as equipment such as a sweeping robot, monitoring equipment, AR / VR, service robots, and personal verification equipment. As long as it is equipped with the multipath mitigation system, this application will not go into details.

[0117] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0118] The above embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention.

Claims

1. A multipath detection method, characterized in that: Includes steps: Obtaining the original data collected by the TOF module and calculating the point cloud map corresponding to the original data; The point cloud image is divided into sub-point cloud planes according to the neighborhood window of each pixel, so as to fit the plane equation of each sub-point cloud plane; Based on the three-dimensional coordinates of the current pixel and the center point and normal vector of the sub-point cloud plane corresponding to the current pixel, a measurement light vector and a reflection light vector of the current pixel are obtained; Calculating the multipath influence amount of the current pixel based on the measured light vector and the reflected light vector; as well as It is determined whether the multipath influence amount is greater than an influence amount threshold. If so, the current pixel is determined to be an abnormal pixel; if not, the current pixel is determined to be a normal pixel.

2. The multipath detection method according to claim 1, characterized in that: The step of obtaining the measurement light vector and the reflection light vector of the current pixel based on the three-dimensional coordinates of the current pixel and the center point and the normal vector of the sub-point cloud plane corresponding to the current pixel comprises the steps of: A three-dimensional coordinate system is constructed with the optical center of the lens of the TOF module as the origin; Extracting the three-dimensional coordinates of the current pixel as a measurement light vector of the current pixel; Extracting the three-dimensional coordinates of the center point of the sub-point cloud plane corresponding to the current pixel and the normal vector of the sub-point cloud plane to construct an incident plane; as well as Based on the measured light vector and the incident plane, the reflected light vector of the current pixel is calculated.

3. The multipath detection method according to claim 1 or 2, characterized in that: The step of calculating the multipath influence amount of the current pixel based on the measured light vector and the reflected light vector comprises the steps of: Based on the plane equation of the sub-point cloud plane corresponding to the current pixel and the measurement light vector, acquiring the amplitude, phase and distance of the direct path corresponding to the current pixel; Based on the plane equations of the reflected light vector and other sub-point cloud planes, the amplitude, phase and distance of the indirect path corresponding to the current pixel are obtained; as well as The direct path and the indirect path are vector-superimposed to obtain the multipath influence amount of the current pixel.

4. The multipath detection method according to claim 3, characterized in that: The step of obtaining the amplitude, phase and distance of the indirect path corresponding to the current pixel based on the plane equations of the reflected light vector and other sub-point cloud planes comprises the steps of: Calculate the intersection point between the reflected light vector and other plane equations as the reflection impact point to obtain the three-dimensional coordinates of the reflection impact point; Calculate the distance between the reflection impact point and the center point of the sub-point cloud plane corresponding to the current pixel; Extract the amplitude and distance of the reflection impact point; Calculate the distance of the indirect path corresponding to the current pixel through the indirect distance calculation model; Calculate the phase of the indirect path corresponding to the current pixel through the indirect phase calculation model; as well as The amplitude of the indirect path corresponding to the current pixel is calculated through the indirect amplitude calculation model.

5. The multipath detection method according to claim 4, characterized in that: The indirect distance calculation model is d F =(d0+d1+d3) / 2, where d F is the distance of the indirect path corresponding to the current pixel; d0 is the straight-line distance between the current pixel and the center point of the sub-point cloud plane corresponding to the current pixel; d1 is the straight-line distance between the reflection influence point and the center point; d3 is the straight-line distance between the reflection influence point and the current pixel.

6. The multipath detection method according to claim 4, characterized in that: The indirect phase calculation model is ψ F =(4πfd F ) / c, where ψ F is the phase of the indirect path corresponding to the current pixel; f is the focal length of the lens of the TOF module; d F is the distance of the indirect path corresponding to the current pixel; c is the speed of light.

7. The multipath detection method according to claim 4, characterized in that: The indirect amplitude calculation model is A F =(A Q d3 2 ) / d F 2 , where A F is the amplitude of the indirect path corresponding to the current pixel; d3 is the straight-line distance between the reflection impact point and the current pixel; d F is the distance of the indirect path corresponding to the current pixel.

8. A multipath mitigation method, characterized in that: Includes steps: By using the multipath detection method as described in any one of claims 1 to 7, abnormal pixels and normal pixels in the raw data collected by the TOF module are detected; and The abnormal pixels are filtered out to retain the normal pixels and obtain multipath interference reduction data.

9. A multipath detection system, characterized in that: comprising communicatively connected to each other: A data acquisition module is used to acquire the original data collected by the TOF module and calculate a point cloud map corresponding to the original data; A plane fitting module is used to divide the point cloud image into sub-point cloud planes according to the neighborhood window of each pixel, so as to fit the plane equation of each sub-point cloud plane; A vector acquisition module, used for obtaining a measurement light vector and a reflection light vector of the current pixel based on the three-dimensional coordinates of the current pixel and the center point and normal vector of the sub-point cloud plane corresponding to the current pixel; An influence amount calculation module, used for calculating the multipath influence amount of the current pixel based on the measured light vector and the reflected light vector; as well as The threshold judgment module is used to judge whether the multipath influence amount is greater than the influence amount threshold. If so, the current pixel is determined to be an abnormal pixel; if not, the current pixel is determined to be a normal pixel.

10. The multipath detection system according to claim 9, characterized in that: The vector acquisition module includes: a coordinate system construction module, which is communicatively connected to each other, and is used to construct a three-dimensional coordinate system with the optical center of the lens of the TOF module as the origin; a measurement vector extraction module, which is used to extract the three-dimensional coordinates of the current pixel as the measurement light vector of the current pixel; and an incident plane construction module, which is used to extract the three-dimensional coordinates of the center point of the sub-point cloud plane corresponding to the current pixel and the normal vector of the sub-point cloud plane to construct the incident plane; And a reflection vector calculation module is used to calculate the reflection light vector of the current pixel based on the measured light vector and the incident plane.

11. The multipath detection system according to claim 9 or 10, characterized in that: The influence amount calculation module includes: a direct path acquisition module, which is communicatively connected to each other, and is used to acquire the amplitude, phase and distance of the direct path corresponding to the current pixel based on the plane equation of the sub-point cloud plane corresponding to the current pixel and the measured light vector; and an indirect path acquisition module, which is used to acquire the amplitude, phase and distance of the indirect path corresponding to the current pixel based on the reflected light vector and the plane equations of other sub-point cloud planes; And a vector superposition module is used to perform vector superposition on the direct path and the indirect path to obtain the multipath influence amount of the current pixel.

12. The multipath detection system according to claim 11, characterized in that: The indirect path acquisition module includes: an influence point calculation module, which is communicatively connected to each other, and is used to calculate the intersection point between the reflection light vector and other plane equations as the reflection influence point to obtain the three-dimensional coordinates of the reflection influence point; a center distance calculation module, which is used to calculate the distance between the reflection influence point and the center point of the sub-point cloud plane corresponding to the current pixel; and a reflection distance extraction module, which is used to extract the amplitude and distance of the reflection influence point; An indirect distance calculation module, used to calculate the distance of the indirect path corresponding to the current pixel through an indirect distance calculation model; An indirect phase calculation module, used to calculate the phase of the indirect path corresponding to the current pixel through an indirect phase calculation model; And an indirect amplitude calculation module calculates the amplitude of the indirect path corresponding to the current pixel through an indirect amplitude calculation model.

13. A multipath mitigation system, characterized in that: comprising communicatively connected to each other: The multipath detection system as claimed in any one of claims 9 to 12, used to detect abnormal pixels and normal pixels in raw data collected by a TOF module; and The filtering module is used to filter out the abnormal pixels to retain the normal pixels and obtain multipath interference reduction data.

14. An electronic device, characterized in that include: TOF module; and The multipath mitigation system of claim 13, wherein the multipath mitigation system is communicatively connected to the TOF module.

Images