Method, device and equipment for generating point cloud data

Abstract

The invention discloses a point cloud data generation method, device and equipment, and the device comprises an electromagnetic wave transmitting module which is used for transmitting electromagnetic waves; the beam scanning module is used for deflecting electromagnetic waves to scan a target object along set directions, and the set directions comprise a first direction and a second direction; the echo receiving module is used for receiving echo signals reflected by electromagnetic waves scanning the target object, the echo signals comprise forward echo signals and reverse echo signals, the forward echo signals refer to echo signals reflected by electromagnetic waves scanning the target object in the first direction, and the reverse echo signals refer to echo signals reflected by electromagnetic waves scanning the target object in the second direction. The reverse echo signal refers to an echo signal reflected by an electromagnetic wave which scans the target object along a second direction; and the point cloud generation module is used for generating point cloud data of the target object according to the forward echo signal and the reverse echo signal, and the point cloud data indicates the distance between the device and the target object.

Description

[0001] This application is a divisional application. The original application has the application number 202411135929.5 and the original application date is August 16, 2024. The entire contents of the original application are incorporated herein by reference. Technical Field

[0002] This application relates to the field of radar sensors, and more particularly to a method, apparatus, and device for generating point cloud data. Background Technology

[0003] Multi-sensor fusion integrates information obtained from different sensors, avoiding the limitations and uncertainties of individual sensor perception, forming a more comprehensive perception and recognition of the environment or target, improving the system's external perception capability, and is widely used in fields such as fault detection, remote sensing, human health monitoring, robotic systems, human-computer interaction, target recognition and tracking, simultaneous localization and mapping, and advanced driver assistance systems.

[0004] Sensors are crucial for a system to perceive the external world. The main sensor combinations include radar-camera, camera-LiDAR, and radar-camera-LiDAR. Cameras accurately capture contour, texture, and color distribution information, aiding in the classification and identification of different targets. Millimeter-wave radar and LiDAR can obtain three-dimensional data of targets; compared to two-dimensional data, three-dimensional data can reconstruct the true spatial environment to the greatest extent possible.

[0005] Radar-visual fusion refers to the fusion of image and point cloud data to reconstruct a 3D spatial environment. The main drawback of existing radar-visual fusion methods is that while camera-output images have high resolution, radar / LiDAR-output point clouds are sparse, resulting in a significant resolution gap between the point cloud and the image. Therefore, the point cloud density needs further improvement. Summary of the Invention

[0006] This application provides a method, apparatus, and device for generating point cloud data, which can be used to solve the problem of fuzzy distance, thereby generating high-density point clouds.

[0007] In a first aspect, this application provides a detection device, which includes: (1) an electromagnetic wave emitting module for emitting electromagnetic waves; (2) a beam scanning module for deflecting the electromagnetic waves to scan a target object along a set direction, the set direction including a first direction and a second direction, such as scanning and irradiating the target object along the first direction, and after scanning in the first direction, scanning and irradiating in the second direction; (3) an echo receiving module for receiving the echo signal reflected by the electromagnetic waves scanned to the target object, the echo signal including a forward echo signal and a reverse echo signal, wherein the forward echo signal refers to the echo signal reflected by the electromagnetic waves scanned to the target object along the first direction, and the reverse echo signal refers to the echo signal reflected by the electromagnetic waves scanned to the target object along the second direction; (4) a point cloud generation module for generating point cloud data of the target object based on the forward echo signal and the reverse echo signal, the point cloud data indicating the depth distance between the device and the target object.

[0008] The above design provides a new method for generating point cloud data by scanning the target in both forward and reverse directions and generating point cloud data based on the forward and reverse echo signals. Instead of calculating depth and distance solely based on the flight time of the echo signals, this method provides a new approach to point cloud data generation. The difference between the forward and reverse echo signals is independent of the pulse modulation frequency, thus eliminating the ambiguity distance problem caused by the modulation frequency. This overcomes the limitation of ambiguity distance and can improve the density of point cloud data generated by a single-transmitter, single-receiver detector.

[0009] In one possible design, when the point cloud generation module generates point cloud data of the target object based on the forward and reverse echo signals, it is specifically used to: determine the positional deviation of the target object based on the forward and reverse echo signals, and generate the first point cloud data based on the positional deviation of the target object.

[0010] Through the above design, the positional deviation of the target object is determined based on the forward and reverse echo signals, and point cloud data is generated based on this positional deviation. Since this positional deviation is related to the depth distance from the device 10 to the target and is independent of the modulation frequency of the electromagnetic wave, there is no ambiguity distance problem caused by the modulation frequency, thus overcoming the ambiguity distance limitation and obtaining high-density point cloud data.

[0011] In one possible design, when the point cloud generation module generates point cloud data of the target object based on the forward and reverse echo signals, it is specifically used to: determine the time difference based on the forward and reverse echo signals, and generate the first point cloud data based on the time difference.

[0012] The above design generates point cloud data based on the time difference between the forward and reverse echo signals. This time difference is related to the depth distance between the device 10 and the target, and is independent of the modulation frequency of the electromagnetic wave. It avoids the ambiguity distance problem caused by the modulation frequency, thus overcoming the ambiguity distance limitation and obtaining high-density point cloud data.

[0013] In one possible design, the device further includes an image generation module; the image generation module is used to generate a target image based on the echo signal received by the echo receiving module, the target image including the target object.

[0014] Through the above design, the detection device outputs both point cloud data and images. Compared with sensor combinations such as cameras and radar, the detection device can realize a true radar-vision integrated machine.

[0015] In one possible design, the device also includes an image generation module; The image generation module is used to generate a forward scan image based on the forward echo signal and a reverse scan image based on the reverse echo signal. Both the forward scan image and the reverse scan image include the target object. The positional deviation is determined based on a first position and a second position. The first position is the position of the target object in the forward scan image, and the second position is the position of the target object in the reverse scan image.

[0016] Through the above design, forward and reverse scan images are obtained by scanning the target in both forward and reverse directions. The position deviation of the target in the forward and reverse scan images is calculated by the depth distance from the target to the calculation device 10. Since this position deviation is related to the depth distance to the target and is independent of the modulation frequency of the electromagnetic wave, there is no ambiguity distance problem caused by the modulation frequency, thus overcoming the ambiguity distance limitation and obtaining high-density point cloud data.

[0017] In one possible design, the echo receiving module is also used to generate a depth signal corresponding to each echo signal, the depth signal being used to indicate the distance between the device and the target object; when generating point cloud data of the target object, the point cloud generation module is specifically used to generate second point cloud data based on the first point cloud data and at least one depth signal, the second point cloud data having a higher precision than the first point cloud data.

[0018] Through the above design, by combining the depth signal and the first point cloud data, high-precision, high-density, and high-accuracy second point cloud data can be obtained, further improving the accuracy of the high-density point cloud output by the detection device.

[0019] In one possible design, the echo receiving module is further configured to generate a grayscale signal corresponding to each echo signal based on the signal strength of at least one echo signal, the grayscale signal being used to indicate the brightness of the scanned object; the forward scan image is generated based on the grayscale signal corresponding to each of the at least one forward echo signals, and the reverse scan image is generated based on the grayscale signal corresponding to each of the at least one reverse echo signals.

[0020] In one possible design, the echo receiving module is further configured to generate a depth signal corresponding to at least one echo signal, the depth signal being used to indicate the distance between the device and the target object; the forward scan image is generated based on the depth signal corresponding to each of the at least one forward echo signal, and the reverse scan image is generated based on the depth signal corresponding to each of the at least one reverse echo signal.

[0021] The above design provides multiple ways to generate images, improving flexibility and the diversity of applicable scenarios.

[0022] In one possible design, the beam scanning module is also used to generate a reference signal, which indicates the scanning direction and / or scanning position; the image generation module, when generating forward and reverse scanning images, is specifically used for: A pixel mapping relationship is generated based on a reference signal. The pixel mapping relationship is used to indicate the echo signal corresponding to each pixel in a frame of an image. The value of each pixel is determined based on the echo signal corresponding to the pixel. The forward scan image includes the pixels in a frame of an image that correspond to the forward echo signal, and the reverse scan image includes the pixels in a frame of an image that correspond to the reverse echo signal.

[0023] In one possible design, the positional deviation of the target object is determined by performing instance segmentation on the forward scan image and the reverse scan image to obtain the first target object instance in the forward scan image and the second target object instance in the reverse scan image. The positional deviation of the target object is the distance between the first position of the first target object instance in the forward scan image and the second position of the second target object instance in the reverse scan image.

[0024] The above design provides a method for generating instance (semantic) level point cloud data.

[0025] In one possible design, the positional deviation of the target object is determined in the following way: Feature extraction is performed on the forward scan image to obtain the feature corresponding to each pixel among the multiple pixels included in the forward scan image. The feature includes a feature used to indicate the target object. Feature extraction is also performed on the reverse scan image to obtain the feature corresponding to each pixel among the multiple pixels included in the reverse scan image. The feature includes a feature used to indicate the target object. The positional deviation of the target object includes the distance between the position of the pixel corresponding to the first feature in the forward scan image and the position of the pixel corresponding to the first feature in the reverse scan image. The first feature is any feature belonging to the target object.

[0026] The above design provides a method for generating pixel-level point cloud data.

[0027] In one possible design, the target image is a frame determined based on the echo signal; or

[0028] The target image is a forward-scanned image, which is an image generated based on the forward echo signal in a single frame; or, The target image is a reverse-scanned image, and the forward-scanned image is an image generated based on the reverse echo signal in a single frame; or, The target image is determined based on the forward scan image and the reverse scan image.

[0029] Secondly, this application provides a method for generating point cloud data. The method includes: acquiring echo information, the echo information including information on echo signals reflected by scanning a target object, wherein the scanning direction includes a first direction and a second direction; the echo signals include forward echo signals and reverse echo signals, the forward echo signal referring to the echo signal reflected by electromagnetic waves scanning the target object along the first direction, and the reverse echo signal referring to the echo signal reflected by electromagnetic waves scanning the target object along the second direction; generating point cloud data of the target object based on the forward echo signals and reverse echo signals, wherein the point cloud data indicates the distance between the device and the target object.

[0030] In one possible design, the point cloud data of the target object is generated based on the forward and reverse echo signals, including: The positional deviation of the target object is determined based on the forward and reverse echo signals, and the first point cloud data is generated based on the positional deviation of the target object.

[0031] In one possible design, the point cloud data of the target object is generated based on the forward and reverse echo signals, including: The time difference is determined based on the forward and reverse echo signals, and the first point cloud data is generated based on the time difference.

[0032] In one possible design, the method also includes: A target image is generated based on the received echo signal, and the target image includes the target object.

[0033] In one possible design, the method also includes: A forward scan image is generated based on the forward echo signal, and a reverse scan image is generated based on the reverse echo signal. Both the forward scan image and the reverse scan image include the target object. The positional deviation is determined based on the first position and the second position. The first position is the position of the target object in the forward scan image, and the second position is the position of the target object in the reverse scan image.

[0034] In one possible design, the method also includes: Generate a depth signal corresponding to each echo signal; the depth signal is used to indicate the distance between the device and the target object. The point cloud data of the target object is generated based on the forward and reverse echo signals, including: A second point cloud is generated based on the first point cloud data and at least one depth signal. The accuracy of the second point cloud data is higher than that of the first point cloud data.

[0035] In one possible design, the forward scan image is generated based on the grayscale signal corresponding to each forward echo signal in at least one forward echo signal, and the reverse scan image is generated based on the grayscale signal corresponding to each reverse echo signal in at least one reverse echo signal.

[0036] In one possible design, the forward scan image is generated based on the depth signal corresponding to each of the at least one forward echo signals, and the reverse scan image is generated based on the depth signal corresponding to each of the at least one reverse echo signals.

[0037] In one possible design, the positional deviation of the target object is determined in the following way: Instance segmentation is performed on the forward scan image and the reverse scan image to obtain the first target object instance in the forward scan image and the second target object instance in the reverse scan image; The positional deviation of the target object is the distance between the first position of the first target object instance in the forward scan image and the second position of the second target object instance in the reverse scan image.

[0038] In one possible design, the positional deviation of the target object is determined in the following way: Feature extraction is performed on the forward scan image to obtain the features corresponding to each pixel in the multiple pixels included in the forward scan image. The features include features used to indicate the target object. Feature extraction is performed on the reverse scan image to obtain the features corresponding to each pixel in the multiple pixels included in the reverse scan image. The features include features used to indicate the target object. The positional deviation of the target object includes the distance between the position of the pixel corresponding to the first feature in the forward scan image and the position of the pixel corresponding to the first feature in the reverse scan image. The first feature is any feature belonging to the target object.

[0039] In one possible design, the target image is a frame of image determined based on the echo signal; or the target image is a forward scan image, which is an image generated from a frame of image based on the forward echo signal; or, the target image is a reverse scan image, which is an image generated from a frame of image based on the reverse echo signal; or, the target image is an image determined based on both the forward scan image and the reverse scan image.

[0040] Thirdly, this application provides a vehicle including the detection device of the first aspect of this application and any possible design of the first aspect.

[0041] Fourthly, this application provides a camera, including the detection device of the first aspect of this application and any possible design of the first aspect.

[0042] Fifthly, this application provides a terminal device, including the detection device of the first aspect of this application and any possible design of the first aspect. The terminal device includes robots, drones, wearable smart devices, vehicle-mounted terminal devices, etc.

[0043] In a sixth aspect, this application provides an electronic device including at least one processor, at least one memory, a display screen, and a transceiver; wherein the one or more memories store one or more computer programs, the one or more computer programs including instructions that, when executed by the one or more processors, cause the electronic device to perform the technical solutions of the second aspect of this application and any possible design of the second aspect.

[0044] The seventh aspect provides another electronic device, the electronic device comprising: a module / unit for performing the methods of the second aspect or any possible design of the second aspect; these modules / units may be implemented in hardware or by hardware executing corresponding software.

[0045] The eighth aspect provides a chip coupled to a memory in an electronic device for calling a computer program stored in the memory and executing the technical solutions of the second aspect of this application and any possible design of the second aspect.

[0046] A ninth aspect provides a computer-readable storage medium comprising a computer program that, when executed on an electronic device, causes the electronic device to perform the technical solutions described in the second aspect above and any possible design of the second aspect.

[0047] The tenth aspect provides a computer program comprising instructions that, when executed on a computer, cause the computer to perform the technical solutions described in the second aspect above and any possible design of the second aspect.

[0048] For the beneficial effects described in the second to tenth aspects above, please refer to the description of the beneficial effects in the first aspect above, which will not be repeated here. Attached Figure Description

[0049] Figure 1 This is one of the existing technical solutions; Figure 2 This is a second existing technical solution; Figure 3 This is a third existing technical solution; Figure 4 This is a schematic diagram of a distance detection scenario; Figure 5 A schematic diagram illustrating the fuzzy distance problem; Figure 6 This is a schematic diagram of the structure of a detection device 10 provided in an embodiment of this application; Figure 7 A schematic diagram of a scanning direction provided in an embodiment of this application; Figure 8 A schematic diagram of a single-transmitter, single-receiver structure of a detection device 10 provided in an embodiment of this application; Figure 9 A schematic diagram of a multi-transmitter, multi-receiver structure of a detection device 10 provided in an embodiment of this application; Figure 10 A flowchart illustrating a point cloud data generation method provided in an embodiment of this application; Figure 11A A schematic diagram of scanning parameters provided in an embodiment of this application; Figure 11B A schematic diagram of a scanning path provided for an embodiment of this application; Figure 12 This is a schematic diagram of another scanning path provided in an embodiment of this application; Figure 13A This is a schematic diagram illustrating a scenario of positional deviation provided in an embodiment of this application. Figure 13B This application provides a schematic diagram of a process for generating high-density point cloud data. Figure 13C This is a flowchart illustrating an image generation method provided in an embodiment of this application. Figure 14 This is a schematic diagram of echo signal mapping provided in an embodiment of this application; Figure 15 This is a schematic diagram of an image generation process provided in an embodiment of this application; Figure 16 A schematic diagram of a front and back image provided for an embodiment of this application; Figure 17 This is a flowchart illustrating an image processing method provided in an embodiment of this application. Figure 18 A flowchart illustrating another image processing method provided in this application embodiment; Figure 19A This application provides a schematic diagram of a process for generating high-precision point cloud data in an embodiment of the present application. Figure 19B This is a schematic diagram of the structure of an electromagnetic wave transmitting module 11 provided in an embodiment of this application; Figure 20 This is a schematic diagram of the structure of a beam scanning module 12 provided in an embodiment of this application; Figure 21 This is a schematic diagram of another beam scanning module 12 provided in an embodiment of this application; Figure 22 This is a schematic diagram of the structure of an echo receiving module 13 provided in an embodiment of this application; Figure 23 This is a schematic diagram of an echo signal processing flow provided in an embodiment of this application; Figure 24 This is a schematic diagram of another detection device provided in an embodiment of this application. Detailed Implementation

[0050] First, some terms used in the embodiments of this application will be explained to facilitate understanding by those skilled in the art.

[0051] 1. Radar / Radio Detection and Ranging (RADAR) is used to directionally transmit electromagnetic energy into space. By analyzing the electromagnetic waves reflected by objects in space, the direction, height, speed, and relative distance of those objects can be calculated, and the shape of the objects can also be detected.

[0052] 2. LiDAR (Light Laser Detection and Ranging) is short for laser detection and ranging system. Compared with traditional sensors such as cameras and ultrasonic sensors, LiDAR has the advantages of high measurement accuracy, fast response speed, and strong anti-interference ability. LiDAR has been widely used in the fields of intelligent driving and autonomous driving.

[0053] When a lidar is in operation, it emits electromagnetic waves toward a target object (which can be a person, object, or animal, and can be referred to as the target). After receiving the echo signal reflected back from the target object, the lidar processes the electromagnetic waves and the echo signal to obtain a point cloud.

[0054] 3. Point clouds are sampling points obtained after a detection device probes an object. Each point cloud corresponds to a set of three-dimensional coordinates (x, y, z). z is the depth distance between the detection device and the object.

[0055] The following introduces several existing technical solutions for improving point cloud density.

[0056] Option 1: One of the options for receiving and sending more data; See Figure 1 As shown, in Figure 1 In this array, ALx is an onboard array that emits light, while RLx is a photodetector used to receive the echoes of electromagnetic waves reflected from the surface of an object. ALx can emit multiple electromagnetic beams simultaneously, thereby increasing point cloud density.

[0057] Option Two: The second option is to receive and send more. See Figure 2 As shown, multiple transceiver modules are used to increase point cloud density. Scheme 2 is similar to Scheme 1, both increasing the number of transceiver modules to improve point cloud density. The difference is that in Scheme 2, the multiple transceiver modules are responsible for different view fields, while in Scheme 1, the multiple transceiver modules are in the same view field.

[0058] The disadvantages of Scheme 1 and Scheme 2 include: multiple transmitters and receivers require a large number of light sources and detectors or multiple optical paths, resulting in a large number of components, which will increase material and assembly costs, leading to higher costs, and the products are larger in size and have limited application scenarios.

[0059] Option 3: Multiple collections via face-to-face delivery; Another approach to improving point cloud density using lidar is flash lidar, such as... Figure 3 As shown, a surface light source illuminates the target, and a surface array detector measures the distance. The density of the point cloud in a flash lidar depends on the density of the surface array detector; increasing the number of pixels in the surface array detector can improve the point cloud density. However, this approach is limited by the size constraints of the surface array detector elements, and the current point cloud density remains relatively low.

[0060] Option 4: Increase the electromagnetic wave repetition rate.

[0061] like Figure 4 As shown, point cloud density can be increased by increasing the laser repetition frequency, that is, the number of laser pulses M emitted per unit time. One pulse signal corresponds to the emission of one electromagnetic wave. It can be understood that for a single-receiver, single-transmitter module, the more points emitted per unit time, the greater the point cloud density.

[0062] However, when using Scheme 4 to improve point cloud density, a distance ambiguity problem arises, leading to a shortened radar measurement range. For example... Figure 5 As shown, for pulse radar, to avoid aliasing, the time interval between the echo signal and the transmitted pulse is shorter than the pulse transmission time interval. When the repetition frequency of the transmitted pulse increases, the pulse period, that is, the pulse transmission time interval, shortens. At this time, when the distance to the target (such as a person, vehicle, or object) is greater than the maximum distance corresponding to the pulse repetition period, the target echo does not fall within the current period, i.e., aliasing occurs. The distance measured at this time is not the true distance, and is called fuzzy distance.

[0063] As those skilled in the art will know, the modulation frequency of electromagnetic waves is related to point cloud density and detection distance. A higher modulation frequency results in higher point cloud density, but a shorter maximum detection distance to the surrounding environment. This is because, to prevent aliasing, the next electromagnetic wave should only be emitted after receiving the echo reflected from the electromagnetic wave after it has traveled a certain detection distance. Understandably, the greater the detection distance, the longer the flight time of the electromagnetic wave / echo. Therefore, the modulation frequency of the electromagnetic wave must usually be selected based on the actual environment and cannot be arbitrarily changed, resulting in the point cloud density being limited by the ambiguity distance and unable to be further improved.

[0064] The technical solution provided in this application will be introduced next.

[0065] This application provides a detection device that overcomes the limitations of ambiguous distances and achieves high-density point cloud data output based on a single-transmitter, single-receiver module. Optionally, the device can also output images. Compared to a combination of hardware devices that output images from a camera and point cloud data from sensors such as radar, this application can realize a true integrated radar and vision sensor.

[0066] The types of detection devices may include, but are not limited to: radar sensors, laser sensors, cameras, etc.

[0067] In one application scenario, the detection device is used in vehicles, such as in-vehicle navigation systems and autonomous driving systems. It can be used to generate images of the vehicle while measuring the distance between the vehicle and surrounding objects, thus helping the vehicle avoid obstacles.

[0068] In one application scenario, the detection device is used in a smart terminal, which includes, but is not limited to, mobile phones, wearable devices (such as watches, bracelets, etc.), augmented reality (AR) devices, virtual reality (VR) devices, etc.

[0069] In one application scenario, the detection device is used in monitoring equipment, such as security monitoring and electronic traffic police. The generated point cloud data and images can be used to assist traffic police in determining responsibility.

[0070] In one application scenario, this detection device is used in robots, such as medical robotic arms, handling robots, production robots, and entertainment robots, to detect the distance between the robot and surrounding objects, assisting the robot in avoiding obstacles and positioning. It can also be applied in machine vision scenarios for robots to identify materials.

[0071] In one application scenario, the detection device is used in aerospace equipment, such as airplanes and drones, and can be specifically used in aerospace navigation systems.

[0072] Figure 6 This is a schematic diagram of a detection device provided in an embodiment of this application. Figure 6 As shown, the detection device 10 includes an electromagnetic wave transmitting module 11, a beam scanning module 12, an echo receiving module 13, an image generating module 14, a point cloud generating module 15, and a clock control module 16.

[0073] 1) Electromagnetic wave transmitting module 11, used to transmit electromagnetic waves. The electromagnetic waves may include, but are not limited to, laser sources, radio frequency sources, or terahertz sources. The technical solution provided in this application can solve the problem of fuzzy distance limitation. Therefore, the modulation frequency (or transmission frequency) of the electromagnetic waves used by module 11 can be set based on requirements such as image resolution and point cloud density, thereby generating high-density point clouds.

[0074] 2) Beam scanning module 12 is used to deflect electromagnetic waves to scan an object in a first direction and a second direction. The first and second directions are opposite directions; for example, the first direction is forward and the second direction is reverse. See also... Figure 7 As shown in (a), scanning horizontally from left to right of the target is defined as positive, and scanning from right to left is defined as negative. For example, scanning from top to bottom is defined as positive, and scanning from bottom to top is defined as negative. Alternatively, it can be any other angular direction, such as... Figure 7 As shown in (c), etc., any two directions being opposite to each other are applicable to the embodiments of this application, and no specific limitation is made. For ease of explanation, the following description uses the horizontal and vertical directions as examples.

[0075] In one hardware implementation example, the beam scanning module 12 can deflect the electromagnetic beam incident from the electromagnetic wave emitting module 11 via a scanning mirror, thereby changing the emission direction of the electromagnetic beam. In hardware, the scanning mirror is driven and can rotate horizontally and vertically. Horizontal rotation allows the emitted electromagnetic wave to be deflected and scanned in the forward or reverse direction to illuminate the target. Vertical rotation allows the electromagnetic wave to illuminate different rows for scanning. If the vertical direction remains unchanged, the beam scanning module 12 will repeatedly scan the same row back and forth.

[0076] Based on this, the beam scanning module 12 can also be used to output forward signals, reverse signals, and angle signals. The forward signal indicates that the scanning direction is forward, and the reverse signal indicates that the scanning direction is reverse. The angle signal can include the horizontal rotation angle (denoted as angle signal x) and the vertical rotation angle (denoted as angle signal y). The forward and reverse signals can be determined based on the angle signals or based on the aforementioned driving trigger signals. For example, the trigger signals include a signal that triggers forward rotation (i.e., the forward signal) and a signal that triggers reverse rotation (i.e., the reverse signal). For instance, when the drive rotates forward, it causes the scanning mirror to rotate forward in the horizontal direction, at which time the electromagnetic wave scans the target in the forward direction. When the drive rotates in reverse, it causes the scanning mirror to rotate in the reverse direction in the horizontal direction, at which time the electromagnetic wave scans the target in the reverse direction. This will be explained below and will not be repeated here.

[0077] 3) Echo receiving module 13 is used to receive the echo signal reflected by the electromagnetic waves scanned to the object. In one example, the echo signal may carry information such as intensity. Module 13 can extract the grayscale signal from the intensity carried by the echo signal. The grayscale signal is used to reflect the reflectivity (brightness) of the object.

[0078] The echo receiving module 13 can also be used to generate a depth signal, which indicates the depth distance between the device 10 and the target. For example, the depth distance can be calculated based on the time-of-flight of the echo signal. It should be understood that because aliasing may occur when the electromagnetic wave transmitting module 11 uses a higher modulation frequency, the depth distance calculated here may be an ambiguous distance, that is, the accuracy of the depth distance calculated here is low.

[0079] 4) Image generation module 14, used to generate a target image based on the echo signal received by echo receiving module 13. The target image contains the scanned object, such as the target object.

[0080] 5) Point cloud generation module 15, used to generate point cloud data of the target object. Specifically, the point cloud data of the target object includes one or more point clouds. Each point cloud includes the depth distance between the detection device 10 and the sampling point of the target object. The depth distance is a non-ambiguous distance determined based on the forward echo signal and the reverse echo signal received by the echo receiving module 13.

[0081] 6) Clock control module 16, used to output clock signals to each module for control, synchronization, triggering, etc.

[0082] From a hardware perspective, in one example, see [link to hardware documentation]. Figure 8 As shown, the detection device 10 is a single-transmitter, single-receiver detector, meaning the electromagnetic wave transmitting module 11 includes a single transmitter and the echo receiving module 13 includes a single receiver. In another example, the detection device 10 is a multi-transmitter, multi-receiver detector, meaning the electromagnetic wave transmitting module 11 includes multiple transmitters and the echo receiving module 13 includes multiple receivers. The detection device 10 proposed in this application can achieve high-density point cloud output based on single-transmitter, single-receiver operation. In another example, see... Figure 9 As shown, the detection device 10 is a multiple-transmitter, multiple-receiver detector, such as... Figure 9 As shown, the detection device 10 includes multiple transmitters and multiple receivers. It can be understood that N transmitters and N receivers can increase the point cloud density by up to N times compared to single transmitter and single receiver. Therefore, the following description uses single transmitter and single receiver as an example.

[0083] It should be noted that, Figure 6 The structure of the detection device 10 shown is merely an example. Detection devices 10 that may be applicable to embodiments of this application may include relative... Figure 6 It may have more or fewer modules; for example, in another implementation, the detection device 10 does not include the image generation module 14. The specific implementation is not limited.

[0084] The following is applied to Figure 6 Taking the detection device 10 shown as an example, this application provides a method for generating point cloud data.

[0085] Figure 10 This is a flowchart illustrating a method for generating point cloud data according to an embodiment of this application. The method may include the following steps: Step 1001: Use electromagnetic waves to scan the target object in both forward and reverse directions, and receive the echo signal reflected by the target object.

[0086] Specifically, the scanning method may include multiple rounds of scanning, with each round of scanning corresponding to a sub-trajectory. The scanning path formed by multiple rounds of scanning includes the sub-trajectory corresponding to each round of scanning.

[0087] Figure 11BAn example of a serpentine scanning path is shown. Taking one scan cycle as an example, combined with... Figure 11A The scanning parameters shown illustrate a scanning process that may include: performing a first scan in the forward direction, with the scan trajectory being sub-trajectory 1. After completing the first scan, a recovery motion is performed, i.e., the horizontal direction is reversed, and the device moves to the next scan (e.g., angle signal y+1), thus performing a second scan in the reverse direction, with the scan trajectory being sub-trajectory 2. After completing the second scan, the device rotates horizontally forward again, and moves to the next scan (angle signal y+1), thus performing a third scan in the forward direction, with the scan trajectory being sub-trajectory 3. After completing the third scan, the device reverses horizontally, and moves to the next scan (angle signal y+1), thus performing a fourth scan in the reverse direction, with the scan trajectory being sub-trajectory 4. This process continues until one frame of scanning is completed. The number of scan lines (or the number of scan rounds) contained in one frame of scanning can be set.

[0088] visible, Figure 11B The illustration shows a case where the sub-trajectories do not overlap. In this embodiment, the sub-trajectories may or may not overlap. In one example, regarding overlapping sub-trajectories... Figure 12 An exemplary scanning path is shown, in which some sub-trajectories overlap. The scanning process may include: performing a first scan along the positive direction of the first row, the scan trajectory being sub-trajectory 1. After the first scan is completed, the horizontal direction is reversed while the angle signal y remains unchanged, thus performing a second scan along the negative direction of the first row, the scan trajectory being sub-trajectory 2. After completing the second scan, the horizontal direction is turned positive, the angle signal y+1, and a third scan is performed along the positive direction of the second row, the scan trajectory being sub-trajectory 3. After completing the third scan, the horizontal direction is reversed while the angle signal y remains unchanged, thus performing a fourth scan along the negative direction of the second row, the scan trajectory being sub-trajectory 4. And so on. It can be seen that sub-trajectory 1 and sub-trajectory 2 overlap. Sub-trajectory 3 and sub-trajectory 4 overlap. Sub-trajectory 5 and sub-trajectory 6 overlap. And so on. In another example, the scan path includes all overlapping sub-trajectories. For example, in the above... Figure 12 In the example, when multiple scans only include two forward and reverse scans of the same row, all sub-trajectories overlap, for example, all sub-trajectories only include Figure 12 The sub-trajectories 1 and 2, or only sub-trajectories 3 and 4, etc.

[0089] The echo receiving module 13 receives the echo signals reflected from the surface of the target object by electromagnetic waves during the scanning process. Based on the scanning direction, the received echo signals can be divided into forward echo signals and reverse echo signals. Specifically, the forward echo signal refers to the echo signal reflected along the forward scanning electromagnetic wave, and the reverse echo signal refers to the echo signal reflected along the reverse scanning electromagnetic wave. Alternatively, the forward echo signal refers to the echo signal received during the forward scanning process, and the reverse echo signal refers to the echo signal received during the reverse scanning process.

[0090] Step 1002: Generate point cloud data of the target object based on the forward echo signal and the reverse echo signal. The point cloud data includes the depth distance D (unambiguous distance) between the device 10 and the target object.

[0091] This application provides several methods for determining unambiguous distances, which are described below: Method 1: Determined based on the positional deviation of the target object.

[0092] This method determines the positional deviation of the target object by using forward and reverse echo signals.

[0093] First see Figure 13A As shown, let the electromagnetic wave scan the target object in the positive direction (e.g. Figure 13A (The gray squares in the image represent the target object). Electromagnetic waves scan the target object in the reverse direction. Figure 13A The black dots represent electromagnetic waves, and the white circles represent echo signals. See [link / reference]. Figure 13A As shown, the echo signal deviates in the positive and negative directions due to the flight time (e.g., Δt). Therefore, the position deviation of the target object can be determined based on the positive and negative echo signals.

[0094] Combination Figure 13B Understanding the methods for determining the positional deviation of a target object involves the following steps: Step 1002-1: Generate a forward scan image based on the forward echo signal, and generate a reverse scan image based on the reverse echo signal.

[0095] In one implementation, the image generation module 14 generates a first image based on the echo signal. The first image includes a forward scan image and a reverse scan image, wherein the forward scan image is generated based on the forward echo signal, and the reverse scan image is generated based on the reverse echo signal. The generation methods of the first image, the forward scan image, and the reverse scan image are the same; the image generation method is described here using the first image as an example.

[0096] In one implementation: a first image is generated based on the grayscale signal of the echo signal, the first image being a grayscale map used to indicate the brightness of an object.

[0097] For example, the image generation module 14 first rearranges the received echo signals into a frame of two-dimensional image based on the pulse time series, and then generates a first image based on the grayscale signal of each echo signal.

[0098] See Figure 13C As shown, the method for generating the first image may include: Step 1301: Map the pulse signals one by one to the echo signals received by the echo receiving module 12 based on the pulse time series.

[0099] like Figure 14 As shown in (a), each black dot represents a transmitted pulse signal (or electromagnetic wave), and the white circle represents a received echo signal. The echo signals are aligned with the preceding adjacent pulse signal. It is worth noting that due to ambiguity issues, there may be many aliased echo signals; for example, if two or more echo signals are received within a single transmission interval. If aliasing occurs, one echo signal can be discarded, such as retaining the echo signal closest to the pulse signal for alignment. The ☒ in the figure indicates that there is no echo signal at that location.

[0100] The row and column of each echo signal can be determined based on the angle signal or the scan trajectory signal. Figure 14 (b) exemplarily illustrates the rearranged echo signals. The rearranged echo signals are numbered according to row and column numbers, generating an index value for each echo signal. This index value indicates the position of the echo signal in the two-dimensional image, for example... d 11 This indicates the first echo signal in the first row, i.e., the first echo signal on sub-track 1. d 12 This indicates the second echo signal in the first row. d 1m This represents the m-th echo signal in the first row. d 21 This indicates the first echo signal in the second row, i.e., the first echo signal on sub-track 2. d 22 This indicates the second echo signal in the second row. d 2m represents the m-th echo signal in the 2nd row. Wherein, d nm In this context, n represents the row number and m represents the column number.

[0101] Step 1302: Grid the spatial location indicated by the scanning parameters and integerize the coordinate values ​​to generate an image array.

[0102] For example, the number of scan rows is determined based on the angle signal y to determine the number of pixel rows in a 2D image, and the number of pixel columns in a 2D image is determined based on a pulse time series. The image array includes the integer coordinates of each pixel, as shown in [see...]. Figure 15 As shown in (a), the image array includes the integer coordinates of each pixel.

[0103] Step 1303: Assign the corresponding echo signal to all pixels (or coordinate points) based on the index value of the echo signal.

[0104] For example, see Figure 15 As shown in (b), the echo signal is assigned to the corresponding pixel position according to the index value. For example, suppose the index value of the echo signal indicates the position in the first row and first column (e.g., ...). d 11 If the indicated position is in the first row and second column (e.g., ...), then the echo signal is assigned to the pixel corresponding to the coordinate point (0, 0). d 12 If the echo signal is (0,1), then the echo signal is assigned to the pixel corresponding to the coordinate point (0,1). And so on.

[0105] Step 1304: Generate the first image based on the grayscale signal of the echo signal corresponding to each pixel.

[0106] See Figure 15 (c) The first image includes a forward scan image (see [link]). Figure 16 See (a) and the reverse scan image. Figure 16 (as shown in (b)). The forward scan image is generated from the forward echo signal, and the reverse scan image is generated from the reverse echo signal.

[0107] pass Figure 15 It can be seen that in the first image, the target object generated based on the positive echo signal deviates in the positive direction, and the target object generated based on the negative echo signal deviates in the negative direction.

[0108] It should be noted that step 1304 uses the generation of a complete image frame from all echo signals as an example. In an alternative method, a forward scan image can be generated based solely on the forward echo signal, or a reverse scan image can be generated based solely on the reverse echo signal. Furthermore, the above description uses the generation of a forward scan image from angle signals and pulse time series as an example; in some alternative methods, the angle signal can be replaced with a forward signal or a reverse signal. Additionally, Figure 15 The method of compiling the coordinate values ​​of each pixel shown is only an example and is not limited to any specific method.

[0109] In another implementation: a forward scan image is generated based on the depth signal of the forward echo signal.

[0110] In the above embodiments, replacing the grayscale signal of the echo signal corresponding to each pixel with the depth signal of the echo signal can generate another first image. Since the depth signal is used to indicate the depth distance from the device 10 to the target object, the first image generated based on the depth signal is used to indicate the 3D shape of the target object, or the first image is used to reflect part or all of the outer contour of the target object. For example, if the detection device 10 is applied to a vehicle, the device 10 can be used to detect and scan a vehicle traveling in front. The first image generated in this case may be the 3D shape of the rear of the vehicle in front, or part or all of the outer contour of the rear of the vehicle in front. It can be understood that if the entire rear of the vehicle is scanned, the first image displays the entire outer contour of the rear. If only part of the rear is scanned, the first image displays only part of the outer contour of the rear.

[0111] Step 1002-2: Determine the positional deviation of the target object between the forward scan image and the reverse scan image.

[0112] based on Figure 15 As shown in (c), both the forward scan image and the reverse scan image include the target object. Based on the position of the target object in the forward scan image (denoted as the first position) and the position of the target object in the reverse scan image (denoted as the second position), the position deviation of the target object can be determined.

[0113] Step 1002-3: Determine the depth distance D (non-fuzzy distance) based on the positional deviation of the target object.

[0114] In one example, the position offset is calculated in pixels, using the number of pixels that the target object is spaced between the forward and reverse scan images.

[0115] The depth distance D is calculated based on the positional deviation of the target object and the pixel mapping relationship of the detection device 10. The pixel mapping relationship indicates the correspondence between one or more pixels and distances. For example, one pixel mapping relationship indicates the depth distance corresponding to one pixel, such as the depth distance corresponding to a unit pixel, which can be actually measured. Based on this, the depth distance D corresponding to the positional deviation can be calculated according to the number of pixels corresponding to the positional deviation of the target object and the depth distance corresponding to a unit pixel. Another example is a pixel mapping relationship indicating the correspondence between multiple pixel values ​​and multiple depth distances; the depth distance D corresponding to the number of pixels (pixel values) corresponding to the positional deviation of the target object is found from this pixel mapping relationship.

[0116] Based on this, point cloud data can be generated, including the depth distance of the target object. As mentioned earlier, a point cloud includes three-dimensional coordinates (x, y, z). In this application, the x-coordinate can be an angle signal x, the y-coordinate can be an angle signal y, and the z-coordinate can be the depth distance D. Figure 11AIt is understood that the angle signal x and angle signal y can be determined according to the reception time of the echo signal or the transmission time of the pulse signal corresponding to the echo signal.

[0117] The following describes two point cloud data generation methods with different granularities provided in the embodiments of this application.

[0118] In one implementation: semantic-level point cloud data is generated.

[0119] See Figure 17 As shown, the generation method may include the following steps: Step 1701: Perform instance segmentation on the objects in the forward scan image and the reverse scan image.

[0120] Specifically, instance segmentation can be based on shape segmentation, which yields segmented objects with the same shape in both the forward and reverse scanned images. Alternatively, it can be based on feature segmentation, which yields segmented objects with the same features in both the forward and reverse scanned images. The instance segmentation method can be based on existing or future instance segmentation algorithms / models, which will not be elaborated upon here.

[0121] Step 1702: Perform disparity calculation on the same segmented objects in the forward scan image and the reverse scan image to obtain the pixel deviation of the target object.

[0122] by Figure 15 Taking (c) as an example, disparity calculation is performed on the target object in the forward and reverse scan images to obtain the instance-level pixel deviation of the target object. This pixel deviation includes the displacement / distance in the x-direction. For example, based on the coordinate values ​​of the same point in the forward scan image and the coordinate values ​​in the reverse scan image, the displacement deviation of the two points in the x-direction is calculated, that is, the difference in the x-values ​​of the two coordinate points, and this value is used as the pixel deviation of the target object. If the x-coordinate value of a vertex of the target object in the forward scan image (denoted as the first value) and the x-coordinate value of the center point of the target object in the reverse scan image (denoted as the second value) are taken, such as the first value being 16 and the second value being 10, then the pixel deviation is the difference between the two, i.e., 6 (unit: pixels). This pixel deviation is used as the positional deviation of the target object.

[0123] It should be understood that this example uses a single point. In this embodiment, the positional deviation of the target object can also be determined based on the pixel deviation of multiple identical points of the target object. For example, multiple pairs of identical points can be taken in the front and back images in the manner described above, and multiple pixel deviations can be calculated based on the x-coordinate value of each pair of identical points. The positional deviation of the target object can be calculated based on these multiple pixel deviations, such as the average value of the multiple pixel deviations.

[0124] Here, the position deviation is determined by the displacement deviation in the x-direction. This application embodiment also supports representing the position deviation by other distance differences, such as the displacement offset in the y-direction or the straight-line distance between two points.

[0125] Step 1703: Obtain the depth distance D of different segmented objects based on the positional deviation. See the description above; it will not be repeated here.

[0126] In another implementation: generate pixel-level point cloud data.

[0127] See Figure 18 As shown, the generation method may include the following steps: Step 1801: Extract feature maps from the forward scan image and the reverse scan image to obtain the feature map of the target object in the forward scan image and the feature map of the target object in the reverse scan image.

[0128] Step 1802: Generate the mapping relationship between pixel coordinates and feature maps at different granularities.

[0129] This mapping indicates the coordinates included in the feature map of the target object in the forward scan image, and the coordinates included in the feature map of the target object in the reverse scan image.

[0130] Step 1803: Calculate the disparity of all pixels belonging to the feature map of the target object to obtain the pixel deviation corresponding to each pixel.

[0131] Taking a pair of pixels as an example, based on the x-coordinate value of a pixel on the feature map of the target object in the forward scan image and the x-coordinate value of a pixel at the same position on the feature map of the target object in the reverse scan image, the difference between these two x-coordinate values ​​is taken as the pixel deviation corresponding to the pair of pixels.

[0132] Step 1804: Generate the corresponding depth distance D based on the pixel deviation of each pair of pixels in the feature map of the target object in the front and back images, which includes some or all of the pixels.

[0133] Based on this, point cloud data can be generated, including the depth distance D corresponding to multiple sampling points on the target object. This results in pixel-level point cloud data. The calculation method is described above and will not be repeated here.

[0134] The previous article introduced a method for determining point cloud data based on the positional deviation of a target object. Next, we will introduce another method for generating point cloud data of a target object.

[0135] It should be noted that the depth distance corresponding to a unit pixel may be the same or different in different detection devices. This application does not limit this.

[0136] Through the above design, forward and reverse scan images are obtained by scanning the target in both forward and reverse directions. The positional deviation of the target in the forward and reverse scan images is used to calculate the depth distance from the target to the detection device 10. Since this positional deviation is related to the distance from the detection device 10 to the target object and is independent of the modulation frequency of the electromagnetic wave, there is no ambiguity distance problem caused by the modulation frequency. This overcomes the ambiguity distance limitation and can improve the density of point cloud data generated by a single-transmitter, single-receiver detector.

[0137] Method 2: Determined based on the time difference between the forward and reverse echo signals.

[0138] Similar to the method described earlier for calculating point cloud data based on positional offset, semantic-level or pixel-level temporal differences can also be determined. Figure 15 Taking (c) as an example, based on the reception time t1 of the echo signal corresponding to a pixel of the target object in the forward scan image and the reception time t2 of the echo signal corresponding to a pixel of the target object in the reverse scan image, the time difference δt = t1 - t2 is determined. Based on this time difference δt and the speed of light c, the depth distance D of the target object is calculated as D = 1 / 2δt × c. This example can be understood as instance-level point cloud data calculated based on the time difference. Based on the pixel-level time difference, the depth distance D corresponding to each pixel included in the target object can be calculated, i.e., pixel-level point cloud data. See the relevant description above, which will not be repeated here.

[0139] Through the above design, the depth distance D is calculated based on the image to obtain high-density point cloud data, which is then used for computational supplementation sensing to improve the point cloud acquisition rate, overcome the distance ambiguity limitation, and realize high-speed, high-density point cloud acquisition of distant target objects. Furthermore, it achieves a high point cloud sampling rate with fewer hardware resources, resulting in lower costs, a smaller single-transmitter single-receiver detector, fewer application scenario limitations, and a wider range of applicable spaces.

[0140] For ease of explanation, the point cloud data determined in step 1003 will be referred to as the first point cloud data.

[0141] Step 1003: Determine the second point cloud data based on the first point cloud data and the depth signal generated by the echo generation module 13.

[0142] It should be understood that the echo receiving module 13 itself has a ranging function, so the generated depth signal has high accuracy. However, due to the possibility of aliasing caused by the electromagnetic wave transmitting module 11 using a high modulation frequency, the depth distance measured by the echo receiving module 13 may be an ambiguous distance, i.e., its accuracy is low. The depth distance D generated in this application has higher accuracy than the depth signal generated by the echo receiving module 13. Therefore, in combination with... Figure 19AUnderstandably, this application proposes to generate high-accuracy, high-precision, high-density point cloud data (referred to as second point cloud data) by combining the high-density first point cloud data generated by the above method and the fine-precision point cloud data generated based on the depth signal.

[0143] For example, the method for generating second point cloud data based on first point cloud data and fine-precision point cloud data is described below: The depth distance D of the target point cloud in the first point cloud data and the depth distance of the target point cloud in the fine-precision point cloud data are fused to obtain a fused depth distance, which is then used as the z-coordinate of the target point cloud. As another example, the fusion method may include: adding the high-order byte of the depth distance D of the target point cloud to the low-order byte of the depth distance of the target point cloud in the fine-precision point cloud data. For example, if the depth distance D of the target point cloud is 132m, and the depth signal indicates that the depth distance of the target point cloud is 4.52m, then the fused depth distance could be 134.52m. 134.52m is then used as the final output depth distance of the target point cloud. It should be noted that the fusion method described here is only illustrative; other fusion methods may also be used, such as fusing 132m and 4.52m to obtain 132.45, etc. Specific details will not be elaborated further.

[0144] Through the above design, by combining the depth signal and the first point cloud data, high-accuracy, high-precision, and high-density second point cloud data can be obtained, further improving the accuracy of the high-density point cloud output by the point cloud generation module 15.

[0145] Step 1004: Generate and output the target image, which includes the target object.

[0146] Image generation module 14 generates and outputs the target image. In one example, the target image is a forward-scanned image. In another example, the target image is a reverse-scanned image. In yet another example, the target image is a combination of a forward-scanned image and a reverse-scanned image, as described above. Figure 15 The image shown in (c). Alternatively, the target image comprises two frames: a forward scan image and a reverse scan image. In one example, the target image is a single frame after the position of the target object has been corrected using both the forward and reverse scan images. For instance, based on the center point of the target object in the forward scan image and the center point of the target object in the reverse scan image, the target object is moved as a whole to the midpoint of the line connecting the two center points, thereby correcting the position of the target object in the first image, and the corrected first image is output as the target image.

[0147] In this way, the detection device 10 outputs both point cloud data and images, realizing a true integrated radar and vision machine.

[0148] It should be noted that: (1) Step 1003 is an optional step and is not a mandatory step. The point cloud data output by the point cloud generation module 15 in this embodiment of the application can be first point cloud data or second point cloud data.

[0149] (2) Step 1004 is an optional step and is not a mandatory step. For example, when the detection device 10 does not include the image generation module 14, it may not output an image.

[0150] (3) The above examples all use straight line segments as examples of sub-trajectories. The sub-trajectories in the embodiments of this application can also be other shapes such as wavy lines or broken lines. This application does not limit them.

[0151] (4) The above examples are all illustrated by scanning forward first and then scanning backward. The embodiments of this application also support scanning backward first and then scanning forward, and there is no specific limitation.

[0152] The following describes a hardware control method for some modules in the detection device 10, in conjunction with the above-described method embodiments.

[0153] See Figure 19B As shown, in one example, the electromagnetic wave emitting module 11 includes a laser driver, a laser diode, and an optical path. The laser driver outputs an electrical signal of a certain frequency to control the laser diode to emit light. The laser diode emits laser light, which is then input to the beam scanning module 12 after passing through the optical path. The laser source can be pulse-modulated, sinusoidally modulated, randomly encoded modulated, or continuously output. The optical path is used to collimate the laser light emitted by the laser diode. In a specific embodiment, the optical path can be a collimating lens or a collimating lens group. That is, the laser source can be collimated spatially or coupled into the optical chip.

[0154] The beam scanning module 12 can use deflected laser beams, such as MEMS scanning mirrors, voice coil scanning mirrors, and optical phased arrays (OPA), to scan target objects from both forward and reverse directions. For example, see [link to MEMS scanning mirror documentation]. Figure 20 As shown, the scanning mirror drive rotates the MEMS scanning mirror, thereby deflecting the incident light beam to scan the target object from the front, and similarly, to scan the target object from the back. Alternatively, see [link to documentation]. Figure 21 As shown, the scanning beam can also be re-emitted after passing through a suitable shaping optical path. Figure 20 and Figure 21 The example shown is a single-transmitter, single-receiver module. When the detection device 10 includes a multi-transmitter, multi-receiver module, it can also scan multiple beams simultaneously.

[0155] Echo receiver module 13: Used to receive echo signals. In one design, using... Figure 21For example, the echo signal can first pass through the shaping optical path and the MEMS scanning mirror before entering the echo receiving module 13. In another design, the echo signal can also directly enter the echo receiving module 13 without passing through any components. See Figure 22 As shown in (a), the echo signal entering the echo receiving module 13 can be focused onto the photodetector by a lens. Alternatively, see Figure 22 As shown in (b), the echo signal directly illuminates the photodetector. The photodetector is matched with the light source, and the photodetector includes, but is not limited to, a photoelectric detector (PD), an avalanche photodiode (APD), a single-photon avalanche diode (SPAD), a photomultiplier tube (PMT), and a multi-pixel photon counter (MPPC). Optionally, a filter of the corresponding wavelength band can be added in front of the photodetector. Specific details are not limited.

[0156] Figure 23 This is an exemplary signal processing diagram of an echo receiving module 13 according to an embodiment of this application. The echo signal illuminates a photodetector, which converts the received echo into photocurrent. The photocurrent is then converted into an echo voltage signal by a TIA circuit. The echo voltage signal undergoes two processing stages: an ADC circuit extracts the reflectivity intensity carried by the echo signal to obtain a grayscale signal; and a time-to-digital converter (TDC) circuit outputs a time vector, which is then converted into a depth distance (possibly an ambiguous distance) to obtain a depth signal indicating that depth distance.

[0157] This application also provides a detection device 2400. Figure 24 This is a schematic diagram of a detection device 2400 provided in an embodiment of this application. The detection device 2400 can be applied to vehicles, cameras, or various terminal devices mentioned above. (See also...) Figure 24 As shown, the detection device 2400 may include a transceiver 2401 and a chip 2402.

[0158] Functionally, in one alternative design, the transceiver 2401 is used to perform... Figure 6 The electromagnetic wave transmitting module 11 and echo receiving module 13 in the detection device 10 shown perform their respective functions. Optionally, the transceiver 2401 can also perform the functions of the beam scanning module 12. The chip 2402 is used to perform... Figure 6The point cloud generation module 15 in the detection device 10 shown performs the functions of the image generation module 14, and the optional chip 2402 can also perform the functions of the image generation module 14. In terms of hardware, in one design, the transceiver 2401 may include an electromagnetic wave transmitting module 11, a beam scanning module 12, and an echo receiving module 13. The chip 2402 may be, but is not limited to, any one or more of the following processors: a central processing unit (CPU), a data processing unit (DPU), a system-on-chip (SoC), a field-programmable gate array (FPGA), a graphics processing unit (GPU), and an application-specific integrated circuit (ASIC).

[0159] Based on the above embodiments, this application also provides a computer program that, when run on a computer, causes the computer to perform... Figure 10 The illustrated embodiment provides a point cloud data generation method.

[0160] Based on the above embodiments, this application also provides a computer-readable storage medium storing a computer program, which, when executed by a computer, causes the computer to perform... Figure 10 The illustrated method embodiment provides a method for generating point cloud data. The storage medium can be any available medium that can be accessed by a computer. For example, but not limited to, a computer-readable medium can include RAM, ROM, EEPROM, CD-ROM or other optical disc storage, disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code having an instruction or data structure form and accessible by a computer.

[0161] Based on the above embodiments, this application also provides a chip, which is used to read a computer program stored in a memory and implement... Figure 10 The illustrated embodiment provides a point cloud data generation method.

[0162] Based on the above embodiments, this application provides a chip system including a processor for supporting computer devices to implement... Figure 10The illustrated embodiment provides a point cloud data generation method. In one possible design, the chip system further includes a memory for storing necessary programs and data for the computer device. The chip system may consist of chips or may include chips and other discrete components.

[0163] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0164] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0165] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0166] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0167] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the scope of protection of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A detection device, characterized in that, The device includes: Electromagnetic wave transmitting module, used to transmit electromagnetic waves; A beam scanning module is used to deflect electromagnetic waves to scan a target object along a set direction, the set direction including a first direction and a second direction; An echo receiving module is used to receive the echo signal reflected by the electromagnetic wave scanned to the target object. The echo signal includes a forward echo signal and a reverse echo signal. The forward echo signal refers to the echo signal reflected by the electromagnetic wave scanned to the target object along the first direction, and the reverse echo signal refers to the echo signal reflected by the electromagnetic wave scanned to the target object along the second direction. A point cloud generation module is used to generate point cloud data of the target object based on the forward echo signal and the reverse echo signal, wherein the point cloud data indicates the distance between the device and the target object.

2. The apparatus as claimed in claim 1, characterized in that, When generating point cloud data of the target object based on the forward echo signal and the reverse echo signal, the point cloud generation module is specifically used for: The positional deviation of the target object is determined based on the forward echo signal and the reverse echo signal, and first point cloud data is generated based on the positional deviation of the target object.

3. The apparatus as described in claim 1, characterized in that, When generating point cloud data of the target object based on the forward echo signal and the reverse echo signal, the point cloud generation module is specifically used for: The time difference is determined based on the forward echo signal and the reverse echo signal, and the first point cloud data is generated based on the time difference.

4. The apparatus according to any one of claims 1-3, characterized in that, The device also includes an image generation module; The image generation module is used to generate a target image based on the echo signal received by the echo receiving module, and the target image includes the target object.

5. The apparatus according to any one of claims 1-4, characterized in that, The device also includes an image generation module; The image generation module is used to generate a forward scan image based on the forward echo signal and a reverse scan image based on the reverse echo signal, wherein both the forward scan image and the reverse scan image include the target object; The positional deviation is determined based on a first position and a second position, wherein the first position is the position of the target object in the forward scan image and the second position is the position of the target object in the reverse scan image.

6. The apparatus as claimed in claim 2 or 3, characterized in that, The echo receiving module is also used to generate a depth signal corresponding to each echo signal, the depth signal being used to indicate the distance between the device and the target object; When generating point cloud data for the target object, the point cloud generation module is specifically used for: A second point cloud data is generated based on the first point cloud data and at least one depth signal, wherein the accuracy of the second point cloud data is higher than that of the first point cloud data.

7. The apparatus as claimed in claim 5, characterized in that, The echo receiving module is also used to generate a grayscale signal corresponding to each echo signal based on the signal strength of at least one echo signal, the grayscale signal being used to indicate the brightness of the scanned object. The forward scan image is generated based on the grayscale signal corresponding to each forward echo signal in at least one forward echo signal, and the reverse scan image is generated based on the grayscale signal corresponding to each reverse echo signal in at least one reverse echo signal.

8. The apparatus as claimed in claim 5, characterized in that, The echo receiving module is further configured to generate at least one depth signal corresponding to an echo signal, the depth signal being used to indicate the distance between the device and the target object; The forward scan image is generated based on the depth signal corresponding to each forward echo signal in at least one forward echo signal, and the reverse scan image is generated based on the depth signal corresponding to each reverse echo signal in at least one reverse echo signal.

9. The apparatus according to any one of claims 5-8, characterized in that, The beam scanning module is also used to generate a reference signal, which is used to indicate the scanning direction and / or scanning position; The image generation module, when generating forward-scanning and reverse-scanning images, is specifically used for: A pixel mapping relationship is generated based on the reference signal. The pixel mapping relationship is used to indicate the echo signal corresponding to each pixel in a frame of an image. The value of each pixel is determined based on the echo signal corresponding to the pixel. The forward scan image includes the pixels in the frame of the image that correspond to the forward echo signal, and the reverse scan image includes the pixels in the frame of the image that correspond to the reverse echo signal.

10. The apparatus according to any one of claims 5-9, characterized in that, The positional deviation of the target object is determined using the following methods: Instance segmentation is performed on the forward scan image and the reverse scan image to obtain a first target object instance in the forward scan image and a second target object instance in the reverse scan image; The positional deviation of the target object is the distance between the first position of the first target object instance in the forward scan image and the second position of the second target object instance in the reverse scan image.

11. The apparatus according to any one of claims 5-9, characterized in that, The positional deviation of the target object is determined using the following methods: Feature extraction is performed on the forward scan image to obtain the features corresponding to each pixel in the plurality of pixels included in the forward scan image, the features including features for indicating the target object; Feature extraction is performed on the reverse scan image to obtain the features corresponding to each pixel in the multiple pixels included in the reverse scan image, the features including features for indicating the target object; The positional deviation of the target object includes the distance between the position of the pixel corresponding to the first feature in the forward scan image and the position of the pixel corresponding to the first feature in the reverse scan image, wherein the first feature is any feature belonging to the target object.

12. The apparatus as claimed in claim 4, characterized in that, The target image is a frame determined based on the echo signal; or The target image is the forward scan image, which is an image generated based on the forward echo signal in one frame of the image; or, The target image is the reverse scan image, and the forward scan image is the image generated based on the reverse echo signal in the frame; or... The target image is an image determined based on the forward scan image and the reverse scan image.

13. A method for generating point cloud data, characterized in that, The method includes: The method involves acquiring echo information, which includes information about the echo signals reflected when scanning a target object. The scanning direction includes a first direction and a second direction. The echo signals include forward echo signals and reverse echo signals. The forward echo signal refers to the echo signal reflected by the electromagnetic waves that scan the target object along the first direction, and the reverse echo signal refers to the echo signal reflected by the electromagnetic waves that scan the target object along the second direction. The point cloud data of the target object is generated based on the forward echo signal and the reverse echo signal, and the point cloud data indicates the distance between the device and the target object.

14. The method as described in claim 13, characterized in that, The step of generating point cloud data of the target object based on the forward echo signal and the reverse echo signal includes: The positional deviation of the target object is determined based on the forward echo signal and the reverse echo signal, and first point cloud data is generated based on the positional deviation of the target object.

15. The method as described in claim 13, characterized in that, The step of generating point cloud data of the target object based on the forward echo signal and the reverse echo signal includes: The time difference is determined based on the forward echo signal and the reverse echo signal, and the first point cloud data is generated based on the time difference.

16. The method according to any one of claims 13-15, characterized in that, The method further includes: A target image is generated based on the received echo signal, the target image including the target object.

17. The method according to any one of claims 13-16, characterized in that, The method further includes: A forward scan image is generated based on the forward echo signal, and a reverse scan image is generated based on the reverse echo signal. Both the forward scan image and the reverse scan image include the target object. The positional deviation is determined based on a first position and a second position, wherein the first position is the position of the target object in the forward scan image and the second position is the position of the target object in the reverse scan image.

18. The method as described in claim 14 or 15, characterized in that, The method further includes: Generate a depth signal corresponding to each echo signal, the depth signal being used to indicate the distance between the device and the target object; The step of generating point cloud data of the target object based on the forward echo signal and the reverse echo signal includes: A second point cloud data is generated based on the first point cloud data and at least one depth signal, wherein the accuracy of the second point cloud data is higher than that of the first point cloud data.

19. The method as described in claim 17, characterized in that, The positional deviation of the target object is determined using the following methods: Instance segmentation is performed on the forward scan image and the reverse scan image to obtain a first target object instance in the forward scan image and a second target object instance in the reverse scan image; The positional deviation of the target object is the distance between the first position of the first target object instance in the forward scan image and the second position of the second target object instance in the reverse scan image.

20. The method as described in claim 17, characterized in that, The positional deviation of the target object is determined using the following methods: Feature extraction is performed on the forward scan image to obtain the features corresponding to each pixel in the plurality of pixels included in the forward scan image, the features including features for indicating the target object; Feature extraction is performed on the reverse scan image to obtain the features corresponding to each pixel in the multiple pixels included in the reverse scan image, the features including features for indicating the target object; The positional deviation of the target object includes the distance between the position of the pixel corresponding to the first feature in the forward scan image and the position of the pixel corresponding to the first feature in the reverse scan image, wherein the first feature is any feature belonging to the target object.

21. The method according to any one of claims 13-20, characterized in that, The target image is a frame determined based on the echo signal; or The target image is the forward scan image, which is an image generated based on the forward echo signal in one frame of the image; or, The target image is the reverse scan image, and the forward scan image is the image generated based on the reverse echo signal in the frame; or... The target image is an image determined based on the forward scan image and the reverse scan image.

22. A vehicle, characterized in that, Includes the detection device as described in any one of claims 1-12.

23. A camera, characterized in that, Includes the detection device as described in any one of claims 1-12.

24. A terminal device, characterized in that, Includes the detection device as described in any one of claims 1-12.

Images