Point cloud data generation method, apparatus, and device

Abstract

A point cloud data generation method, an apparatus, and a device. The apparatus comprises: an electromagnetic wave transmitting module, which is used for transmitting electromagnetic waves; a beam scanning module, which is used for deflecting the electromagnetic waves to scan a target object in set directions, the set directions comprising a first direction and a second direction; an echo receiving module, which is used for receiving echo signals reflected from the target object scanned by the electromagnetic waves, the echo signals comprising a forward echo signal and a reverse echo signal, wherein the forward echo signal refers to an echo signal reflected from the target object scanned by the electromagnetic waves in the first direction, and the reverse echo signal refers to an echo signal reflected from the target object scanned by the electromagnetic waves in the second direction; and a point cloud generation module, which is used for generating point cloud data of the target object on the basis of the forward echo signal and the reverse echo signal, the point cloud data indicating the distance between the apparatus and the target object.

Description

A point cloud data generation method, device and equipment

[0001] Cross-reference to Related Applications

[0002] This application claims priority to the Chinese Patent Application No. 202411135929.5, filed on August 16, 2024, and entitled "A point cloud data generation method, device and equipment", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0003] The present application relates to the field of radar sensors, and in particular to a point cloud data generation method, device and equipment. BACKGROUND

[0004] Multi-sensor fusion comprehensively utilizes information obtained by different sensors, which can avoid the perception limitations and uncertainties of a single sensor, form a more comprehensive perception and recognition of the environment or target, improve the external perception ability of the system, and is widely used in fault detection, remote sensing, human health monitoring, robot systems, human-computer interaction, target recognition and tracking, simultaneous localization and mapping, advanced driving assistance systems and other fields.

[0005] Sensors are the key to the system's perception of the external world, and the main sensor combinations include radar-camera, camera-lidar and radar-camera-lidar. Cameras can accurately capture contour, texture and color distribution information, which helps to classify and identify different targets. Millimeter wave radar and lidar can obtain three-dimensional data of the target, and compared with two-dimensional data, three-dimensional data can maximize the recovery of the real space environment.

[0006] Radar-visual fusion refers to the fusion of image and point cloud data to construct a restored three-dimensional space environment. The main drawback of existing radar-visual fusion is that the resolution of the image output by the camera is high, while the point cloud output by the radar / lidar is sparse, and the resolution gap between the point cloud and the image is relatively large, and the point cloud density needs to be further improved. SUMMARY

[0007] The present application provides a point cloud data generation method, device and equipment, which can be used to solve the problem of fuzzy distance, thereby generating high-density point cloud.

[0008] In a first aspect, the present application provides a detection device, which comprises: (1) an electromagnetic wave emitting module for emitting electromagnetic waves; (2) a beam scanning module for deflecting the electromagnetic waves to scan a target object along a set direction, the set direction including a first direction and a second direction, such as scanning and irradiating the target object along the first direction, and then scanning and irradiating along the second direction after the first direction scanning is completed; (3) a return wave receiving module for receiving return wave signals reflected by the electromagnetic waves scanned to the target object, the return wave signals including forward return wave signals and reverse return wave signals, wherein the forward return wave signals refer to the return wave signals reflected by the electromagnetic waves scanned to the target object along the first direction, and the reverse return wave signals refer to the return wave signals reflected by the electromagnetic waves scanned to the target object along the second direction; (4) a point cloud generation module for generating point cloud data of the target object based on the forward return wave signals and the reverse return wave signals, the point cloud data indicating the depth distance between the device and the target object.

[0009] Through the above design, by scanning the target in the forward and reverse directions, the point cloud data is generated based on the forward return wave signals and the reverse return wave signals, instead of calculating the depth distance by the time of flight of the return wave signals alone, a new point cloud data generation method is provided, wherein the difference between the forward return wave signals and the reverse return wave signals is independent of the pulse modulation frequency, so there is no problem of ambiguous distance caused by the modulation frequency, and the ambiguous distance limit is broken, and the density of the point cloud data generated by the single-transmit-single-receive detector can be improved.

[0010] In a possible design, when generating the point cloud data of the target object based on the forward return wave signals and the reverse return wave signals, the point cloud generation module is specifically configured to: determine a position deviation of the target object based on the forward return wave signals and the reverse return wave signals, and generate first point cloud data based on the position deviation of the target object.

[0011] Through the above design, the position deviation of the target object is determined based on the forward return wave signals and the reverse return wave signals, and the point cloud data is generated based on the position deviation. Since the position deviation is related to the depth distance of the device 10 to the target and is independent of the modulation frequency of the electromagnetic waves, there is no problem of ambiguous distance caused by the modulation frequency, and the ambiguous distance limit is broken, so that high-density point cloud data is obtained.

[0012] In a possible design, when generating the point cloud data of the target object based on the forward return wave signals and the reverse return wave signals, the point cloud generation module is specifically configured to: determine a time difference based on the forward return wave signals and the reverse return wave signals, and generate first point cloud data based on the time difference.

[0013] By the above design, the point cloud data is generated based on the time difference between the forward echo signal and the backward echo signal. The time difference is related to the depth distance of the device 10 to the target, and is irrelevant to the modulation frequency of the electromagnetic wave. There is no ambiguity distance problem caused by the modulation frequency, and the ambiguity distance limit is broken through, so that high-density point cloud data is obtained.

[0014] In a possible design, the device further includes an image generation module; the image generation module is configured to generate a target image based on the echo signals received by the echo receiving module, and the target image includes the target object.

[0015] By the above design, the detection device outputs both point cloud data and images. Compared with a camera-radar sensor combination, the detection device can realize a true radar-vision all-in-one machine.

[0016] In a possible design, the device further includes an image generation module;

[0017] The image generation module is configured to generate a forward scanning image based on the forward echo signal, and generate a backward scanning image based on the backward echo signal, and the forward scanning image and the backward scanning image both include the target object. The position deviation amount is determined according to a first position and a second position, the first position being the position of the target object in the forward scanning image, and the second position being the position of the target object in the backward scanning image.

[0018] By the above design, the forward scanning image and the backward scanning image are obtained by performing forward and backward bidirectional direction scanning on the target. The position deviation amount of the target in the forward scanning image and the backward scanning image is used to calculate the depth distance of the device 10 to the target. Since the position deviation amount is related to the depth distance to the target, and is irrelevant to the modulation frequency of the electromagnetic wave, there is no ambiguity distance problem caused by the modulation frequency, and the ambiguity distance limit is broken through, so that high-density point cloud data is obtained.

[0019] In a possible design, the echo receiving module is further configured to generate a depth signal corresponding to each echo signal, and the depth signal is used to indicate the distance between the device and the target object. When generating the point cloud data of the target object, the point cloud generation module is specifically configured to generate second point cloud data based on the first point cloud data and at least one depth signal, and the accuracy of the second point cloud data is higher than that of the first point cloud data.

[0020] By the above design, the depth signal and the first point cloud data are combined to obtain second point cloud data with high precision, high density and high accuracy, and the precision of the detection device outputting high-density point cloud is further improved.

[0021] In a possible design, the echo receiving module is further configured to generate a gray-scale signal corresponding to each of the at least one echo signal based on a signal strength of the at least one echo signal, where the gray-scale signal is used to indicate a light-dark degree of the scanned object; the forward scan image is generated based on a gray-scale signal corresponding to each of the at least one forward echo signal, and the backward scan image is generated based on a gray-scale signal corresponding to each of the at least one backward echo signal.

[0022] In a possible design, the echo receiving module is further configured to generate a depth signal corresponding to each of the at least one echo signal, where the depth signal is used to indicate a distance between the device and the target object; the forward scan image is generated based on a depth signal corresponding to each of the at least one forward echo signal, and the backward scan image is generated based on a depth signal corresponding to each of the at least one backward echo signal.

[0023] Through the above design, multiple image generation manners are provided, and flexibility and diversity of applicable scenarios are improved.

[0024] In a possible design, the beam scanning module is further configured to generate a reference signal, where the reference signal is used to indicate a scanning direction and / or a scanning position; and the image generation module is specifically configured to:

[0025] generate a pixel mapping relationship based on the reference signal, where the pixel mapping relationship is used to indicate a corresponding echo signal of each pixel in a plurality of pixels included in a frame of image, and a value of each pixel is determined based on the corresponding echo signal of the pixel; and the forward scan image includes pixels corresponding to the forward echo signal in the frame of image, and the backward scan image includes pixels corresponding to the backward echo signal in the frame of image.

[0026] In a possible design, the position deviation of the target object is determined in the following manner: performing instance segmentation on the forward scan image and the backward scan image to obtain a first target object instance in the forward scan image and a second target object instance in the backward scan image.

[0027] The position deviation of the target object is a distance between a first position of the first target object instance in the forward scan image and a second position of the second target object instance in the backward scan image.

[0028] Through the above design, an instance (semantic)-level point cloud data generation manner is provided.

[0029] In a possible design, the position deviation of the target object is determined in the following manner:

[0030] The feature extraction is performed on the forward scanning image to obtain a feature corresponding to each pixel in the plurality of pixels included in the forward scanning image, and the feature includes a feature for indicating the target object; the feature extraction is performed on the reverse scanning image to obtain a feature corresponding to each pixel in the plurality of pixels included in the reverse scanning image, and the feature includes a feature for indicating the target object; and the position deviation of the target object includes a distance between a position of a pixel corresponding to the first feature in the forward scanning image and a position of a pixel corresponding to the first feature in the reverse scanning image, the first feature being any one feature attributed to the target object.

[0031] Through the above design, a pixel-level point cloud data generation manner is provided.

[0032] In a possible design, the target image is a frame of image determined based on the echo signal; or

[0033] The target image is the forward scanning image, and the forward scanning image is an image generated based on the forward echo signal in a frame of image; or

[0034] The target image is the reverse scanning image, and the forward scanning image is an image generated based on the reverse echo signal in a frame of image; or

[0035] The target image is an image determined based on the forward scanning image and the reverse scanning image.

[0036] In a second aspect, the present application provides a point cloud data generation method, which includes: obtaining echo information, the echo information including information of an echo signal reflected by a target object scanned, wherein the scanning direction includes a first direction and a second direction; the echo signal includes a forward echo signal and a reverse echo signal, the forward echo signal being an echo signal reflected by an electromagnetic wave scanning the target object along the first direction, and the reverse echo signal being an echo signal reflected by an electromagnetic wave scanning the target object along the second direction; and generating point cloud data of the target object based on the forward echo signal and the reverse echo signal, the point cloud data indicating a distance between the device and the target object.

[0037] In a possible design, the generating of the point cloud data of the target object based on the forward echo signal and the reverse echo signal includes:

[0038] determining a position deviation of the target object based on the forward echo signal and the reverse echo signal, and generating first point cloud data based on the position deviation of the target object.

[0039] In a possible design, the generating of the point cloud data of the target object based on the forward echo signal and the reverse echo signal includes:

[0040] determining a time difference based on the forward echo signal and the reverse echo signal, and generating first point cloud data based on the time difference.

[0041] In a possible design, the method further includes:

[0042] generating a target image based on the received echo signals, the target image including the target object.

[0043] In a possible design, the method further includes:

[0044] generating a forward scanning image based on the forward echo signals and generating a backward scanning image based on the backward echo signals, the forward scanning image and the backward scanning image both including the target object;

[0045] The position deviation of the target object is determined according to a first position of the target object in the forward scanning image and a second position of the target object in the backward scanning image.

[0046] In a possible design, the method further includes:

[0047] generating a depth signal corresponding to each echo signal, the depth signal being used to indicate a distance between the device and the target object;

[0048] generating point cloud data of the target object based on the forward echo signals and the backward echo signals, including:

[0049] generating second point cloud data based on the first point cloud data and the at least one depth signal, the second point cloud data having a higher accuracy than the first point cloud data.

[0050] In a possible design, the forward scanning image is generated based on a gray-scale signal corresponding to each of the at least one forward echo signal, and the backward scanning image is generated based on a gray-scale signal corresponding to each of the at least one backward echo signal.

[0051] In a possible design, the forward scanning image is generated based on a depth signal corresponding to each of the at least one forward echo signal, and the backward scanning image is generated based on a depth signal corresponding to each of the at least one backward echo signal.

[0052] In a possible design, the position deviation of the target object is determined in the following manner:

[0053] performing instance segmentation on the forward scanning image and the backward scanning image to obtain a first target object instance in the forward scanning image and a second target object instance in the backward scanning image;

[0054] The position deviation of the target object is a distance between a first position of the first target object instance in the forward scanning image and a second position of the second target object instance in the backward scanning image.

[0055] In a possible design, the position deviation of the target object is determined in the following manner:

[0056] The forward scanning image is subjected to feature extraction, to obtain features corresponding to each of pixels included in the forward scanning image, the features including features for indicating the target object;

[0057] The reverse scanning image is subjected to feature extraction, to obtain features corresponding to each of pixels included in the reverse scanning image, the features including features for indicating the target object;

[0058] The position deviation of the target object includes a distance between a position of a pixel corresponding to the first feature in the forward scanning image and a position of a pixel corresponding to the first feature in the reverse scanning image, the first feature being any one feature attributed to the target object.

[0059] In a possible design, the target image is one frame of image determined based on the echo signal; or the target image is the forward scanning image, the forward scanning image being an image generated based on the forward echo signal in one frame of image; or the target image is the reverse scanning image, the forward scanning image being an image generated based on the reverse echo signal in one frame of image; or the target image is an image determined based on the forward scanning image and the reverse scanning image.

[0060] In a third aspect, the present application provides a vehicle, including the detection apparatus in the first aspect of the present application and any possible design of the first aspect.

[0061] In a fourth aspect, the present application provides a camera, including the detection apparatus in the first aspect of the present application and any possible design of the first aspect.

[0062] In a fifth aspect, the present application provides a terminal device, including the detection apparatus in the first aspect of the present application and any possible design of the first aspect. The terminal device includes a robot, a drone, a wearable smart device, a vehicle-mounted terminal device, and the like.

[0063] In a sixth aspect, the present application provides an electronic device, including at least one processor, at least one memory, a display screen, and a transceiver; wherein the one or more memories store one or more computer programs including instructions, when the instructions are executed by the one or more processors, causing the electronic device to perform the technical solutions in the second aspect of the present application and any possible design of the second aspect.

[0064] In a seventh aspect, another electronic device is provided, which includes: a module / unit for performing the method in the second aspect or any possible design of the second aspect; these modules / units can be implemented by hardware, or by hardware executing corresponding software.

[0065] The eighth aspect provides a chip coupled with a memory in an electronic device, used to invoke a computer program stored in the memory and execute the technical solutions of the second aspect and any possible design of the second aspect.

[0066] The ninth aspect provides a computer readable storage medium, including a computer program, when the computer program is run on an electronic device, the electronic device executes the technical solutions of the second aspect and any possible design of the second aspect.

[0067] The tenth aspect provides a computer program, including instructions, when the instructions are run on a computer, the computer executes the technical solutions of the second aspect and any possible design of the second aspect.

[0068] The beneficial effects of the second aspect to the tenth aspect are described above, and the description of the beneficial effects of the first aspect will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0069] Fig. 1 is one of the prior art solutions;

[0070] Fig. 2 is another prior art solution;

[0071] Fig. 3 is a third prior art solution;

[0072] Fig. 4 is a schematic diagram of a distance detection scenario;

[0073] Fig. 5 is a schematic diagram of a fuzzy distance problem;

[0074] Fig. 6 is a schematic diagram of a detection device 10 provided by an embodiment of the present application;

[0075] Fig. 7 is a schematic diagram of a scanning direction provided by an embodiment of the present application;

[0076] Fig. 8 is a schematic diagram of a single-transmit-single-receive structure of a detection device 10 provided by an embodiment of the present application;

[0077] Fig. 9 is a schematic diagram of a multi-transmit-multi-receive structure of a detection device 10 provided by an embodiment of the present application;

[0078] Fig. 10 is a schematic diagram of a flow corresponding to a point cloud data generation method provided by an embodiment of the present application;

[0079] Fig. 11A is a schematic diagram of a scanning parameter provided by an embodiment of the present application;

[0080] Fig. 11B is a schematic diagram of a scanning path provided by an embodiment of the present application;

[0081] FIG. 12 is a schematic diagram of another scanning path according to an embodiment of the present application;

[0082] FIG. 13A is a schematic diagram of a position deviation scenario according to an embodiment of the present application;

[0083] FIG. 13B is a schematic diagram of a process of generating high-density point cloud data according to an embodiment of the present application;

[0084] FIG. 13C is a schematic diagram of a process corresponding to an image generation method according to an embodiment of the present application;

[0085] FIG. 14 is a schematic diagram of echo signal mapping according to an embodiment of the present application;

[0086] FIG. 15 is a schematic diagram of an image generation process according to an embodiment of the present application;

[0087] FIG. 16 is a schematic diagram of a positive image and a negative image according to an embodiment of the present application;

[0088] FIG. 17 is a schematic diagram of a process corresponding to an image processing method according to an embodiment of the present application;

[0089] FIG. 18 is a schematic diagram of a process corresponding to another image processing method according to an embodiment of the present application;

[0090] FIG. 19A is a schematic diagram of a process of generating high-precision point cloud data according to an embodiment of the present application;

[0091] FIG. 19B is a schematic diagram of a structure of an electromagnetic wave transmitting module 11 according to an embodiment of the present application;

[0092] FIG. 20 is a schematic diagram of a structure of a beam scanning module 12 according to an embodiment of the present application;

[0093] FIG. 21 is a schematic diagram of another structure of a beam scanning module 12 according to an embodiment of the present application;

[0094] FIG. 22 is a schematic diagram of a structure of an echo receiving module 13 according to an embodiment of the present application;

[0095] FIG. 23 is a schematic diagram of an echo signal processing process according to an embodiment of the present application;

[0096] FIG. 24 is a schematic diagram of another structure of a detection device according to an embodiment of the present application. DETAILED DESCRIPTION

[0097] First, some terms in the embodiments of the present application are explained to facilitate understanding by those skilled in the art.

[0098] 1. Radar / Radio Detection And Ranging (RADAR), used to emit electromagnetic energy into space in a directional manner, and by the electromagnetic waves reflected by the objects existing in the space, the direction, height, speed, relative distance of the object can be calculated, and the shape of the object can be detected.

[0099] 2. Lidar (light laser detection and ranging), is the abbreviation of laser detection and ranging system. Compared with traditional sensors such as cameras and ultrasonic sensors, lidar has the characteristics of high measurement accuracy, fast response speed and strong anti-interference ability. Lidar has been widely used in the field of intelligent driving and unmanned driving.

[0100] In the working of lidar, electromagnetic waves are emitted to the target object (which can be a person, an object, an animal, which can be referred to as a target). After the lidar receives the echo signal reflected back from the target object, the lidar processes the electromagnetic wave and the echo signal to obtain a point cloud.

[0101] 3. Point cloud, is a sampling point obtained by detecting an object by a detection device. Each point cloud corresponds to a set of three-dimensional coordinates (x, y, z). Z is the depth distance between the detection device and the object.

[0102] Several existing technical solutions for improving the density of point clouds are introduced below.

[0103] Solution one: one of the multiple-transmitting and multiple-receiving solutions;

[0104] Referring to FIG. 1, in FIG. 1, ALx is a board-mounted array light source, and RLx is a light detector for receiving the echo of the electromagnetic wave reflected by the surface of the object. ALX can emit multiple electromagnetic beams at the same time to improve the density of the point cloud.

[0105] Solution two: the second multiple-transmitting and multiple-receiving solution;

[0106] Referring to FIG. 2, multiple transceiver modules are used to improve the density of the point cloud. Solution two and solution one are similar, both of which increase the number of transceiver modules to improve the density of the point cloud. The difference between the two solutions is that the multiple transceiver modules of solution two are responsible for different angle fields, while the multiple transceiver modules of solution one are in the same angle field.

[0107] The disadvantages of solution one and solution two include: multiple-transmitting and multiple-receiving use many light sources and detectors or multiple light paths, the number of components is large, the material cost and assembly cost will increase accordingly, the cost is high, and the product size is large, the application scene is limited.

[0108] Solution three: multiple-receiving surface-transmitting solution;

[0109] Another solution to improve the point cloud density of the laser radar is the flash laser radar as shown in FIG. 3, a surface light source irradiates a target, and a surface array detector measures the distance. The density of the point cloud of the flash laser radar depends on the density of the surface array detector, and the point cloud density can be improved by increasing the number of pixels of the surface array detector. However, this solution is limited by the volume of the elements of the surface array detector, and the point cloud density is still relatively low at present.

[0110] Solution four: improve the electromagnetic wave repetition frequency.

[0111] As shown in FIG. 4, the point cloud density can be improved by increasing the repetition frequency of the laser, that is, the number of laser pulses emitted per unit time M. One pulse signal corresponds to the emission of one electromagnetic wave, and it can be understood that for a single-transmitting and single-receiving module, the more the number of points per unit time, the greater the point cloud density.

[0112] However, when the point cloud density is improved by using solution four, the problem of distance ambiguity will be encountered, which leads to the shortening of the radar measurement distance. As shown in FIG. 5, for a pulse radar, in order to avoid aliasing, the time interval between the echo signal and the transmitted pulse is less than the pulse transmission time interval. When the repetition frequency of the transmitted pulse is increased, the pulse period, that is, the transmission time interval of the pulse, is shortened. At this time, when the distance of the target to be measured (such as a person, a vehicle, or an object) is greater than the maximum distance corresponding to the pulse repetition period, the target echo does not fall within the current period, that is, aliasing occurs, and the distance measured between the target is a non-real distance, which is called ambiguous distance.

[0113] As known by those skilled in the art, the modulation frequency of the electromagnetic wave is related to the point cloud density and the detection distance, and the higher the modulation frequency of the electromagnetic wave, the higher the point cloud density, but the shorter the maximum detection distance of the surrounding environment. This is because, in order to ensure that aliasing does not occur, the next electromagnetic wave should be transmitted after receiving the echo reflected by the electromagnetic wave flying for a detection distance, so that aliasing can be avoided. It can be understood that the longer the detection distance, the longer the flight time of the electromagnetic wave / echo, and therefore, the modulation frequency of the electromagnetic wave cannot be changed at will, which leads to the limitation of the point cloud density by the ambiguous distance and the inability to further improve the point cloud density.

[0114] Next, the technical solutions provided in the present application are introduced.

[0115] The embodiment of the present application provides a detection device which can break through the limitation of the ambiguous distance and realize high-density point cloud data output on the basis of the hardware of a single-transmitting and single-receiving module (single transmitter and single receiver). Optionally, the device can also output images, and compared with the combination of the hardware device of the camera output image + the point cloud data output by the radar sensor, the present application can realize a real radar vision integrated sensor.

[0116] The product type of the detection device can include, but is not limited to, radar sensors, laser sensors, cameras, and the like.

[0117] In an application scenario, the detection device is applied to a vehicle, such as a vehicle navigation system or an unmanned driving system, and can be used to generate a vehicle driving image and measure the distance between the vehicle and surrounding objects to assist the vehicle in avoiding obstacles.

[0118] In an application scenario, the detection device is applied to a smart terminal, which can include, but is not limited to, a mobile phone, a wearable device (such as a watch, a bracelet, and the like), an augmented reality (AR) device, a virtual reality (VR) device, and the like.

[0119] In an application scenario, the detection device is applied to a monitoring device, such as a security monitoring device or an electronic traffic police device, and the generated point cloud data and images can be used to assist the traffic police in determining responsibility.

[0120] In an application scenario, the detection device is applied to a robot, such as a medical robot, a carrying robot, a production robot, an entertainment robot, and the like, and is used to detect the distance between the robot and surrounding objects to assist the robot in avoiding obstacles and positioning, and can also be applied to a machine vision scenario to assist the robot in identifying materials.

[0121] In an application scenario, the detection device is applied to aerospace equipment, such as an airplane or a drone, and can be used in an aerospace navigation system.

[0122] FIG. 6 is a structural schematic diagram of a detection device provided by an embodiment of the present application. As shown in FIG. 6, the detection device 10 includes an electromagnetic wave emitting module 11, a beam scanning module 12, a return wave receiving module 13, an image generating module 14, a point cloud generating module 15, and a clock control module 16.

[0123] 1) The electromagnetic wave emitting module 11 is configured to emit electromagnetic waves. The electromagnetic waves can include, but are not limited to, a laser source, a radio frequency source, or a terahertz source. The technical solution provided by the present application can solve the problem of fuzzy distance limitation, and thus the modulation frequency (or emission frequency) of the electromagnetic waves used by the module 11 can be set based on the requirements of image resolution and point cloud density, so as to generate high-density point clouds.

[0124] 2) a beam scanning module 12 for deflecting the electromagnetic wave to perform a first direction scan and a second direction scan on the object. The first direction and the second direction are opposite directions, for example, the first direction is a forward direction, and the second direction is a reverse direction. For example, referring to (a) of FIG. 7, the forward direction is defined as scanning from the left side to the right side of the target in the horizontal direction, and the reverse direction is defined as scanning from the right side to the left side of the target. For another example, the forward direction is defined as scanning from top to bottom, and the reverse direction is defined as scanning from bottom to top, or the forward direction and the reverse direction can be in any other angle direction, as shown in (c) of FIG. 7, and the like. As long as the two directions are opposite, they are applicable to the embodiments of the present application, and the specific directions are not limited. For ease of description, the horizontal direction and the vertical direction are taken as examples for description below.

[0125] In a hardware implementation example, the beam scanning module 12 can deflect the electromagnetic wave beam incident by the electromagnetic wave emitting module 11 to change the outgoing direction of the electromagnetic wave beam. In hardware, the scanning mirror is driven and controlled to generate horizontal rotation and vertical rotation. The horizontal rotation can realize the deflection of the outgoing electromagnetic wave to scan and irradiate the target in the horizontal direction in the forward direction or the reverse direction. The vertical rotation can change the electromagnetic wave to scan different rows. If the vertical direction remains unchanged, the beam scanning module 12 will repeatedly scan the same row back and forth.

[0126] Based on this, the beam scanning module 12 can also be used to output a forward signal, a reverse signal, and an angle signal. The forward signal can be used to indicate that the scanning direction is the forward direction, and the reverse signal can be used to indicate that the scanning direction is the reverse direction. The angle signal can include a horizontal rotation angle (denoted as angle signal x) and a vertical rotation angle (denoted as angle signal y). The forward signal and the reverse signal can be determined based on the angle signal, or based on the trigger signal of the foregoing driving, such as a signal triggering the driving of the forward rotation (i.e., the forward signal) and a signal triggering the driving of the reverse rotation (i.e., the reverse signal). For example, when the driving of the forward rotation drives the scanning mirror to rotate in the horizontal direction, the electromagnetic wave scans the target in the forward direction. When the driving of the reverse rotation drives the scanning mirror to rotate in the horizontal direction, the electromagnetic wave scans the target in the reverse direction. This will be described below, and will not be described here.

[0127] 3) a return wave receiving module 13 for receiving a return wave signal reflected by the electromagnetic wave scanned to the object. In an example, the return wave signal can carry intensity information, and the return wave receiving module 13 can extract a gray scale signal from the intensity carried by the return wave signal, and the gray scale signal is used to reflect the reflectivity (light and dark) of the object.

[0128] The echo receiving module 13 can also be used to generate a depth signal for indicating a depth distance between the detection device 10 and the target. For example, the depth distance is calculated based on the time of flight of the echo signal. It should be understood that due to aliasing when the electromagnetic wave transmitting module 11 uses a higher modulation frequency, the depth distance calculated here can be an ambiguous distance, i.e., the depth distance calculated here has a lower accuracy.

[0129] 4) The image generating module 14 is used to generate a target image based on the echo signal received by the echo receiving module 13, and the target image contains a scanned object, such as a target object.

[0130] 5) The point cloud generating module 15 is used to generate point cloud data of the target object, and specifically, the point cloud data of the target object includes one or more point clouds, and each point cloud includes a depth distance between the detection device 10 and a sampling point of the target object, and the depth distance is a non-ambiguous distance determined based on the forward echo signal and the backward echo signal received by the echo receiving module 13.

[0131] 6) The clock control module 16 is used to output a clock signal to each module for control, synchronization, triggering, etc.

[0132] From the hardware layer, in one example, referring to FIG. 8, the detection device 10 is a single-transmit-single-receive detector, i.e., the electromagnetic wave transmitting module 11 includes a single transmitter, and the echo receiving module 13 includes a single receiver. In another example, the detection device 10 is a multi-transmit-multi-receive detector, i.e., the electromagnetic wave transmitting module 11 includes multiple transmitters, and the echo receiving module 13 includes multiple receivers. The detection device 10 proposed in the present application can realize high-density point cloud output on the basis of single-transmit-single-receive. In another example, referring to FIG. 9, the detection device 10 is a multi-transmit-multi-receive detector, as shown in FIG. 9, the detection device 10 includes multiple transmitters and multiple receivers. It can be understood that N-transmit-N-receive can increase the point cloud density by a factor of N relative to single-transmit-single-receive. Therefore, the following will be introduced taking single-transmit-single-receive as an example.

[0133] It should be noted that the structure of the detection device 10 shown in FIG. 6 is only an example, and the detection device 10 that can be applied in the embodiments of the present application can include more or fewer modules relative to FIG. 6, for example, in another implementation, the detection device 10 does not include the image generating module 14. Etc., and the specific limitation is not made.

[0134] The following will take the detection device 10 shown in FIG. 6 as an example to introduce a point cloud data generation method provided by the embodiments of the present application.

[0135] FIG. 10 is a flowchart of a point cloud data generation method provided by the embodiments of the present application. The method can include the following steps:

[0136] Step 1001, using electromagnetic waves to scan the target object in forward and reverse directions, and receiving echo signals reflected by the target object.

[0137] Specifically, the scanning manner can include multiple rounds of scanning, each round of scanning corresponding to a sub-track, and a scanning path formed by the multiple rounds of scanning including the sub-tracks corresponding to each round of scanning.

[0138] FIG. 11B exemplarily shows a scanning path in a snake shape. Taking one round of scanning as one row of scanning as an example, in combination with the scanning parameters shown in FIG. 11A, the scanning process can include: performing first row scanning in a forward direction, and the scanning track is sub-track 1. After completing the first row scanning, a return motion is performed, that is, the horizontal direction is reversed, and the next row (for example, the angle signal y+1) is moved to, so as to perform second row scanning in a reverse direction, and the scanning track is sub-track 2. After completing the second row scanning, the horizontal direction is reversed again, and the next row (the angle signal y+1) is moved to, so as to perform third row scanning in a forward direction, and the scanning track is sub-track 3. After completing the third row scanning, the horizontal direction is reversed again, and the next row (the angle signal y+1) is moved to, so as to perform fourth row scanning in a reverse direction, and the scanning track is sub-track 4. In this way, a frame of scanning is completed. The number of scanning rows (or the number of rounds of scanning) included in one frame of scanning can be set.

[0139] It can be seen that FIG. 11B shows a case in which the sub-tracks do not overlap. In the embodiments of the present application, the sub-tracks can overlap or not overlap. For the case in which the sub-tracks overlap, in one example, FIG. 12 exemplarily shows a scanning path in which part of the sub-tracks overlap. The scanning process can include: performing first round scanning in a forward direction along the first row, and the scanning track is sub-track 1. After completing the first round scanning, the horizontal direction is reversed, and the angle signal y is unchanged, so as to perform second round scanning in a reverse direction along the first row, and the scanning track is sub-track 2. After completing the second round scanning, the horizontal direction is reversed, and the angle signal y+1, so as to perform third round scanning in a forward direction along the second row, and the scanning track is sub-track 3. After completing the third round scanning, the horizontal direction is reversed, and the angle signal y is unchanged, so as to perform fourth round scanning in a reverse direction along the second row, and the scanning track is sub-track 4. In this way, a frame of scanning is completed. It can be seen that sub-track 1 and sub-track 2 overlap. Sub-track 3 and sub-track 4 overlap. Sub-track 5 and sub-track 6 overlap. In this way, a frame of scanning is completed. In another example, the scanning path includes sub-tracks that all overlap. For example, in the example of FIG. 12, when the multiple rounds of scanning include only two rounds of forward and reverse scanning for the same row, all the sub-tracks overlap, for example, all the sub-tracks include only sub-track 1 and sub-track 2 in FIG. 12, or only sub-track 3 and sub-track 4, and the like.

[0140] The echo receiving module 13 receives the echo signal reflected by the electromagnetic wave on the surface of the target object during the scanning process, and the received echo signal can be divided into forward echo signal and reverse echo signal according to the scanning direction. Among them, the forward echo signal refers to the echo signal reflected by the forward scanning electromagnetic wave, and the reverse echo signal refers to the echo signal reflected by the reverse scanning electromagnetic wave. Alternatively, the forward echo signal refers to the echo signal received during the forward scanning process, and the reverse echo signal refers to the echo signal received during the reverse scanning process.

[0141] In step 1002, the point cloud data of the target object (denoted as first point cloud data) is generated based on the forward echo signal and the reverse echo signal, and the first point cloud data includes the depth distance D (non-blurring distance) between the detection device 10 and the target object.

[0142] The present application provides various determination methods of non-blurring distance, which are introduced as follows:

[0143] Method one: determination based on the position deviation of the target object.

[0144] This method determines the position deviation of the target object through the forward echo signal and the reverse echo signal.

[0145] Firstly, referring to FIG. 13A, the electromagnetic wave scans the target object (such as the gray block in FIG. 13A representing the target object) in the forward direction, and the electromagnetic wave scans the target object in the reverse direction. The black dots in FIG. 13A represent electromagnetic waves, and the white circles represent echo signals. Referring to FIG. 13A, the echo signals deviate in the forward direction and the reverse direction respectively due to the time of flight (such as Δt), and thus the position deviation of the target object can be determined based on the forward echo signal and the reverse echo signal.

[0146] In combination with FIG. 13B, the method for determining the position deviation of the target object includes the following steps:

[0147] In step 1002-1, a forward scanning image is generated based on the forward echo signal, and a reverse scanning image is generated based on the reverse echo signal.

[0148] In an embodiment, the image generating module 14 generates a first image based on the echo signal, and the first image includes a forward scanning image and a reverse scanning image. The forward scanning image is generated based on the forward echo signal, and the reverse scanning image is generated based on the reverse echo signal. The generation methods of the first image, the forward scanning image and the reverse scanning image are the same, and the image generation method is introduced here by taking the first image as an example.

[0149] In an embodiment, the first image is generated based on the gray signal of the echo signal, and the first image is a gray image for indicating the brightness of the object.

[0150] Exemplarily, the image generation module 14 first rearranges the received echo signals into a two-dimensional image based on the pulse time sequence, and then generates a first image based on the gray-scale signals of each echo signal.

[0151] Referring to FIG. 13C, the method of generating the first image can include:

[0152] Step 1301: Map the echo signals received by the echo receiving module 13 to the pulse signals one by one according to the pulse time sequence.

[0153] As shown in FIG. 14(a), each black dot represents a transmitted pulse signal (or electromagnetic wave), and the white circle represents a received echo signal. The echo signals are aligned according to the previous adjacent pulse signal. It is worth noting that due to the blurring distance problem, there can be many overlapping echo signals, for example, two or more echo signals are received within one transmission interval. If overlapping occurs, one of the echo signals can be discarded, such as the echo signal closest to the pulse signal is retained for alignment. In the figure indicates that there is no echo signal at this position.

[0154] Based on the angle signal or the scanning trajectory signal, the row and column where each echo signal is located can be determined. FIG. 14(b) exemplarily shows the rearranged echo signals. The rearranged echo signals are numbered according to the row number and the column number to generate an index value of each echo signal, which is used to indicate the position of the echo signal in the two-dimensional image, for example, d 11 indicates the first echo signal in the first row, i.e., the first echo signal on the sub-track 1. 12 indicates the second echo signal in the first row, d 1m indicates the mth echo signal in the first row. 21 indicates the first echo signal in the second row, i.e., the first echo signal on the sub-track 2. 22 indicates the second echo signal in the second row, d nm wherein n represents the row number and m represents the column number.

[0155] Step 1302: Grid the spatial position indicated by the scanning parameter, and integerize the coordinate value to generate an image array.

[0156] For example, based on the angle signal y, the number of scanning rows is determined to determine the number of pixel rows of the two-dimensional image, and based on the pulse time sequence, the number of pixel columns of the two-dimensional image is determined. The image array includes the integerized coordinates of each pixel, as shown in FIG. 15(a).

[0157] Step 1303: Assign the corresponding echo signal to the pixel (or coordinate point) based on the index value of the echo signal.

[0158] For example, as shown in (b) of FIG. 15, the echo signals are assigned to the pixels in the corresponding positions according to the index values. For example, if the index value of an echo signal indicates the position of the first row and the first column (as shown in d 11 ), the echo signal is assigned to the pixel corresponding to the coordinate point (0, 0). If the index value indicates the position of the first row and the second column (as shown in d 12 ), the echo signal is assigned to the pixel corresponding to the coordinate point (0, 1). Similarly, the echo signals are assigned to the pixels in the corresponding positions according to the index values.

[0159] Step 1304: generating a first image based on the gray-scale signals of the echo signals corresponding to each pixel.

[0160] As shown in (c) of FIG. 15, the first image includes a forward scanning image (as shown in (a) of FIG. 16) and a reverse scanning image (as shown in (b) of FIG. 16). The forward scanning image is generated based on the forward echo signals, and the reverse scanning image is generated based on the reverse echo signals.

[0161] As shown in FIG. 15, in the first image, the target object generated based on the forward echo signals deviates towards the forward direction, and the target object generated based on the reverse echo signals deviates towards the reverse direction.

[0162] It should be noted that step 1304 is an example of generating a complete image based on all echo signals. In an alternative manner, only the forward scanning image can be generated based on the forward echo signals, or only the reverse scanning image can be generated based on the reverse echo signals. In addition, the above description is an example of generating the forward scanning image based on the angle signals and the pulse time sequence. In some alternative manners, the angle signals can be replaced by the forward signals and the reverse signals. In addition, the coordinate values of the pixels shown in FIG. 15 are only an example, and the specific manner is not limited.

[0163] In another embodiment, the forward scanning image is generated based on the depth signals of the forward echo signals.

[0164] In the above embodiments, the gray scale signal of the echo signal corresponding to each pixel is replaced by the depth signal of the echo signal, another first image can be generated. Since the depth signal is used to indicate the depth distance from the detection device 10 to the target object, the first image generated based on the depth signal is used to indicate the 3D shape of the target object, or the first image is used to reflect part or all of the outer contour of the target object. For example, the detection device 10 is applied in a vehicle, and the detection device 10 can be used to perform a detection scan on a vehicle in front, and the first image generated at this time can be the 3D shape of the tail of the front vehicle, or part or all of the outer contour of the tail of the front vehicle. It can be understood that if the entire tail is scanned, the first image displays the entire outer contour of the tail. If part of the tail is scanned, the first image displays part of the outer contour of the tail.

[0165] Step 1002-2: Determine the position deviation of the target object between the forward scan image and the reverse scan image.

[0166] Based on (c) shown in FIG. 15, both the forward scan image and the reverse scan image include the target object, and based on the position of the target object in the forward scan image (denoted as a first position) and the position of the target object in the reverse scan image (denoted as a second position), the position deviation of the target object can be determined.

[0167] Step 1002-3: Determine the depth distance D (non-blurring distance) based on the position deviation of the target object.

[0168] In an example, the number of pixels between the target object in the forward scan image and the reverse scan image is taken as the position deviation in units of pixels.

[0169] The depth distance D is calculated based on the position deviation of the target object and the pixel mapping relationship of the detection device 10. The pixel mapping relationship indicates the correspondence between one or more pixels and the distance. For example, one pixel mapping relationship indicates the depth distance corresponding to one pixel, for example, the depth distance corresponding to a unit pixel, which can be actually measured. Based on this, the depth distance D corresponding to the position deviation can be calculated according to the number of pixels corresponding to the position deviation of the target object and the depth distance corresponding to a unit pixel. For another example, another pixel mapping relationship indicates the correspondence between a plurality of pixel values and a plurality of depth distances, and the depth distance D corresponding to the number of pixels (pixel value) corresponding to the position deviation of the target object is found from the pixel mapping relationship.

[0170] Based on this, a point cloud data including the depth distance of the target object can be generated. As described above, a point cloud includes three-dimensional coordinates (x, y, z). In this application, the x coordinate can be the angle signal x. The y coordinate can be the angle signal y, and the z coordinate can be the depth distance D. In combination with FIG. 11A, the angle signal x and the angle signal y can be the angle signal x and the angle signal y corresponding to the echo signal determined according to the receiving time of the echo signal or the transmission time of the pulse signal corresponding to the echo signal.

[0171] The following describes two granular point cloud data generation methods provided by embodiments of the application.

[0172] In an implementation, semantic-level point cloud data is generated.

[0173] Referring to FIG. 17, the generation method can include the following steps:

[0174] Step 1701, instance segmentation is performed on the objects in the forward scanning image and the reverse scanning image.

[0175] Specifically, the instance segmentation can be based on shape segmentation, based on which the segmented objects with the same shape in the forward scanning image and the reverse scanning image can be obtained. Or based on feature segmentation, based on which the segmented objects with the same feature in the forward scanning image and the reverse scanning image can be obtained. Wherein, the instance segmentation method can be based on the existing or future possible application of instance segmentation algorithm / model for segmentation execution, which will not be described here.

[0176] Step 1702, parallax calculation is performed on the same segmented objects in the forward scanning image and the reverse scanning image, to obtain the pixel deviation of the target object.

[0177] Taking (c) of FIG. 15 as an example, parallax calculation is performed on the target object in the forward scanning image and the reverse scanning image, to obtain the instance-level pixel deviation of the target object. The pixel deviation includes the displacement amount / distance in the x direction. For example, based on the coordinate values of the same point in the target object in the forward scanning image and the coordinate values in the reverse scanning image, the displacement deviation amount of the two points in the x direction, i.e., the difference between the x values of the two coordinate points, is calculated, and the value is taken as the pixel deviation of the target object. For example, the x coordinate value of a certain vertex of the target object in the forward scanning image (denoted as the first value) and the x coordinate value of the center point of the target object in the reverse scanning image (denoted as the second value) are taken, for example, the first value is 16, and the second value is 10, then the pixel deviation is the difference between the two, i.e., 6 (for example, unit: pixel). The pixel deviation is taken as the position deviation of the target object.

[0178] It should be understood that, as an example of one point, the position deviation of the target object can also be determined based on pixel deviations of multiple same points of the target object, such as taking multiple pairs of same points in the positive and negative images in the manner described above, calculating multiple pixel deviations based on the x coordinate values of each pair of same points, and calculating the position deviation of the target object based on the multiple pixel deviations, such as the average of the multiple pixel deviations.

[0179] Here, the position deviation is determined based on the displacement deviation in the x direction, and the embodiments of the present application also support representing the position deviation by other distance differences, such as the displacement deviation in the y direction or the straight-line distance between two points.

[0180] In step 1703, the depth distance D of the different segmented objects is obtained based on the position deviation. For details, please refer to the description above, which will not be repeated here.

[0181] In another embodiment, pixel-level point cloud data is generated.

[0182] Referring to FIG. 18, the generation method can include the following steps:

[0183] In step 1801, feature maps of the target object in the positive scanning image and the target object in the negative scanning image are obtained by performing feature map extraction on the positive scanning image and the negative scanning image.

[0184] In step 1802, a mapping relationship between the coordinates of the pixel points and the feature maps is generated under different granularities.

[0185] The mapping relationship indicates the coordinates included in the feature map of the target object in the positive scanning image and the coordinates included in the feature map of the target object in the negative scanning image.

[0186] In step 1803, disparity calculation is performed on the (all) pixels belonging to the feature map of the target object to obtain the pixel deviation corresponding to each pixel.

[0187] Taking a pair of pixel points as an example, the pixel deviation corresponding to the pair of pixel points is calculated based on the x coordinate value of a pixel point on the feature map of the target object in the positive scanning image and the x coordinate value of a pixel point at the same position on the feature map of the target object in the negative scanning image.

[0188] In step 1804, the corresponding depth distance D is generated based on the pixel deviation of each pair of pixel points in the part or all of the pixel points included in the feature map of the target object in the positive and negative scanning images.

[0189] Based on this, a kind of point cloud data can be generated, including the depth distance D corresponding to multiple sampling points on the target object. Thus, pixel-level point cloud data is generated. For details of the calculation method, please refer to the description above, which will not be repeated here.

[0190] The above describes a method of determining point cloud data based on the position deviation of the target object. Next, another method of generating point cloud data of the target object is described.

[0191] It should be noted that the depth distance corresponding to a unit pixel in different detection devices can be the same or different. The present application does not limit this.

[0192] Through the above design, the forward scanning image and the reverse scanning image are obtained by bidirectional scanning of the target. The position deviation of the target in the forward scanning image and the reverse scanning image is calculated by the distance from the position deviation calculation device 10 to the target. Since the position deviation is related to the distance from the detection device 10 to the target object and is independent of the modulation frequency of the electromagnetic wave, there is no blur distance problem caused by the modulation frequency, and the blur distance limit is broken, and the density of the point cloud data generated by the single-transmit-single-receive detector can be improved.

[0193] Method two: determining based on the time difference value determined based on the forward echo signal and the reverse echo signal.

[0194] Similar to the above method of calculating point cloud data based on position deviation, semantic-level time difference value or pixel-level time difference value can also be determined. For example, based on the receiving time t1 of the echo signal corresponding to a certain pixel point of the target object in the forward scanning image and the receiving time t2 of the echo signal corresponding to a certain pixel point of the target object in the reverse scanning image, the time difference value δt=t1-t2 is determined, and the depth distance D=1 / 2δt×c of the target object is calculated based on the time difference value δt and the speed of light c. This example can be understood as an instance-level point cloud data calculated based on the time difference value. The depth distance D corresponding to each pixel included in the target object can be calculated based on the pixel-level time difference value, i.e. pixel-level point cloud data. See the related description above, which will not be repeated here.

[0195] Through the above design, the depth distance D is calculated according to the image to obtain high-density point cloud data, to calculate the supplemental perception, improve the point cloud acquisition rate, break through the distance blur limit, and realize high-speed and high-density point cloud acquisition of a long-distance target object; and a high point cloud sampling rate is realized with less hardware resources, the cost is lower, the single-transmit-single-receive detector is smaller in size, the application scene is less limited, and the application space is more extensive.

[0196] For ease of description, the point cloud data determined in step 1003 is referred to as first point cloud data.

[0197] Step 1003, determining second point cloud data based on the first point cloud data and the depth signal generated by the echo generation module 13.

[0198] It should be understood that the echo receiving module 13 itself has a ranging function, so the generated depth signal has high precision, but due to the aliasing problem caused by the higher modulation frequency that the electromagnetic wave transmitting module 11 may take, the depth distance measured by the echo receiving module 13 may be an ambiguous distance, i.e. lower accuracy. The depth distance D generated by the present application has higher accuracy than the depth signal generated by the echo receiving module 13. Therefore, in combination with FIG. 19A, the present application proposes that the high-precision first point cloud data generated in combination with the above method and the fine-precision point cloud data generated based on the depth signal are combined to generate high-accuracy high-precision high-density point cloud data (denoted as second point cloud data).

[0199] For example, the method of generating the second point cloud data based on the first point cloud data and the fine-precision point cloud data is as follows: the depth distance D of the target point cloud in the first point cloud data and the depth distance of the target point cloud in the fine-precision point cloud data are fused to obtain a fused depth distance, and the fused depth distance is taken as the z coordinate of the target point cloud. For another example, the fusion method can include: adding the high-order value of the depth distance D of the target point cloud and the low-order value of the depth distance of the target point cloud in the fine-precision point cloud data, for example, the depth distance D of the target point cloud is 132 m, and the depth signal indicates that the depth distance of the target point cloud is 4.52 m, then the fused depth distance can be 134.52 m. 134.52 m is taken as the final output depth distance of the target point cloud. It should be noted that the fusion method herein is only for illustration, and other fusion methods can also be used, for example, 132 m and 4.52 m are fused to obtain 132.45, etc. Details are not described herein.

[0200] Through the above design, the second point cloud data with high accuracy, high precision and high density can be obtained by combining the depth signal and the first point cloud data, and the precision of the high-density point cloud output by the point cloud generation module 15 is further improved.

[0201] Step 1004, generating and outputting a target image, the target image including a target object.

[0202] The image generation module 14 generates and outputs a target image. In an example, the target image is a forward scanning image. In an example, the target image is a reverse scanning image. In an example, the target image is a combination of the forward scanning image and the reverse scanning image, such as the image shown in (c) of FIG. 15 described above. Alternatively, the target image includes both the forward scanning image and the reverse scanning image. In an example, the target image is one frame of image after correcting the position of the target object using the forward scanning image and the reverse scanning image. For example, based on the center point of the target object in the forward scanning image and the center point of the target object in the reverse scanning image, the target object is moved as a whole to the midpoint of the line connecting the two center points, so as to correct the position of the target object in the first image, and the corrected first image is output as the target image.

[0203] In the above manner, the detection device 10 outputs both point cloud data and images, and realizes a true radar-visual integrated machine.

[0204] It should be noted that:

[0205] (1) Step 1003 is an optional step and is not necessarily performed. The point cloud data output by the point cloud generation module 15 in the embodiment of the application can be first point cloud data or second point cloud data.

[0206] (2) Step 1004 is an optional step and is not necessarily performed. For example, when the detection device 10 does not include the image generation module 14, images can not be output.

[0207] (3) Each of the above examples is illustrated by taking a straight line segment as a sub-trajectory. The sub-trajectory in the embodiment of the application can also be other shapes such as a wavy line or a polyline, which is not limited in the application.

[0208] (4) Each of the above examples is illustrated by taking forward scanning followed by reverse scanning. The embodiment of the application also supports reverse scanning followed by forward scanning, which is not limited in detail.

[0209] The hardware control mode of some modules in the detection device 10 will be introduced as follows in combination with the above method embodiment.

[0210] Referring to FIG. 19B, in an example, the electromagnetic wave emission module 11 includes a laser driver, a laser diode, and an optical path. The laser driver outputs an electrical signal of a certain frequency to control the laser diode to emit light. The laser diode emits laser light, which is input into the beam scanning module 12 after passing through the optical path. The laser source can be pulse-modulated, sinusoidally-modulated, randomly-coded-modulated, or continuously-light-output. The optical path is used to collimate the laser light emitted by the laser diode. In a specific implementation, the optical path can be a collimating lens or a collimating lens group. That is, the laser source can collimate spatial output, or be coupled into an optical chip.

[0211] The beam scanning module 12 can deflect the laser beam by using a MEMS scanning mirror, a voice coil scanning mirror, an optical phased array (OPA), etc. to scan the target object from the forward direction and the reverse direction. Taking the MEMS scanning mirror as an example, referring to FIG. 20, the scanning mirror drive rotates the MEMS scanning mirror, and then deflects the incident light beam to scan the target object from the forward direction, and in the same way, scans the target object from the reverse direction. Alternatively, referring to FIG. 21, the scanning light beam can also be emitted through a properly shaped optical path. FIGS. 20 and 21 are illustrated by taking a single-transmitting and single-receiving module as an example, and when the detection device 10 includes a multiple-receiving and multiple-transmitting module, multiple light beams can also be used for simultaneous scanning.

[0212] The echo receiving module 13 is configured to receive the echo signal. In one design, referring to FIG. 21, the echo signal can first pass through the shaped optical path and the MEMS scanning mirror, and then enter the echo receiving module 13. In another design, the echo signal can also directly enter the echo receiving module 13 without passing through any component. Referring to FIG. 22(a), the echo signal entering the echo receiving module 13 can be focused on the light detector through a lens. Alternatively, referring to FIG. 22(b), the echo signal directly irradiates the light detector. The light detector is matched with the light source, and the light detector includes but is not limited to a photoelectric detector (PD), an avalanche photo diode (APD), a single photon avalanche diode (SPAD), a photo multiplier tube (PMT), a multi-pixel photon counter (MPPC), etc. Optionally, a filter of a corresponding waveband can be added in front of the light detector. The specific implementation is not limited.

[0213] FIG. 23 is a schematic diagram of a signal processing of the echo receiving module 13 according to an example of the present application. The echo signal irradiates the light detector, the light detector converts the received echo into a photoelectric current, the photoelectric current is converted into an echo voltage signal through a TIA circuit, and the echo voltage signal can be processed through two paths. The ADC circuit is configured to extract the reflectivity intensity carried by the echo signal, so as to obtain a gray signal. The time-to-digital converter (TDC) circuit is configured to output a time vector, and convert the time vector into a depth distance (which can be a blur distance), so as to obtain a depth signal for indicating the depth distance.

[0214] The application further provides a detection device 2400. Fig. 24 is a structural schematic diagram of the detection device 2400 according to an embodiment of the application. The detection device 2400 can be applied to the vehicle, the camera or the terminal device in the foregoing.

[0215] In terms of functions, in an optional design, the transceiver 2401 is configured to perform the functions of the electromagnetic wave transmitting module 11 and the echo receiving module 13 in the detection device 10 shown in Fig. 6. Optionally, the transceiver 2401 can also perform the function of the beam scanning module 12. The chip 2402 is configured to perform the function of the point cloud generating module 15 in the detection device 10 shown in Fig. 6. Optionally, the chip 2402 can also perform the function of the image generating module 14. In terms of hardware, in a design, the transceiver 2401 can include the electromagnetic wave transmitting module 11, the beam scanning module 12 and the echo receiving module 13. The chip 2402 can be, but is not limited to, any one or more of the following processors: a central processing unit (CPU) data processing unit (DPU), a system on chip (SOC), a field programmable gate array (FPGA), a graphics processing unit (GPU), an application specific integrated circuit (ASIC) and the like.

[0216] Based on the above embodiments, the application further provides a computer program, which, when executed on a computer, causes the computer to perform the point cloud data generation method provided in the embodiment shown in Fig. 10.

[0217] Based on the above embodiments, the application further provides a computer readable storage medium, which stores a computer program. When the computer program is executed on a computer, the computer program causes the computer to perform the point cloud data generation method provided in the method embodiment shown in Fig. 10. The storage medium can be any available medium accessible by the computer. For example, but not limited to, the computer readable medium can include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage medium or other magnetic storage devices, or any other medium that can be used to carry or store desired program codes in the form of instructions or data structures and can be accessed by the computer.

[0218] Based on the above embodiments, the embodiments of the present application further provide a chip for reading a computer program stored in a memory, and implementing the point cloud data generation method provided by the embodiment shown in FIG. 10.

[0219] Based on the above embodiments, the embodiments of the present application provide a chip system, which comprises a processor for supporting a computer device to implement the point cloud data generation method provided by the embodiment shown in FIG. 10. In a possible design, the chip system further comprises a memory for storing necessary programs and data of the computer device. The chip system can be composed of a chip, or can include the chip and other discrete devices.

[0220] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) containing computer-usable program code.

[0221] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices produce the functions specified in one or more flows in the flowcharts and / or one or more blocks in the block diagrams.

[0222] These computer program instructions can also be stored in a computer-readable memory capable of causing the computer or other programmable data processing devices to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product comprising instruction devices, which implement the functions specified in one or more flows in the flowcharts and / or one or more blocks in the block diagrams.

[0223] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are performed on the computer or other programmable data processing device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more flows in the flowcharts and / or one or more blocks in the block diagrams.

[0224] Obviously, many modifications and variations of the present application are possible in light of the above teachings. It is, therefore, to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A detection device, characterized in that, The device comprises: an electromagnetic wave emitting module configured to emit electromagnetic waves; a beam scanning module configured to deflect the electromagnetic waves to scan a target object in a set direction, the set direction comprising a first direction and a second direction; a return wave receiving module configured to receive return wave signals reflected by the electromagnetic waves scanned to the target object, the return wave signals comprising forward return wave signals and reverse return wave signals, wherein the forward return wave signals refer to return wave signals reflected by the electromagnetic waves scanned to the target object in the first direction, and the reverse return wave signals refer to return wave signals reflected by the electromagnetic waves scanned to the target object in the second direction; a point cloud generating module configured to generate point cloud data of the target object based on the forward return wave signals and the reverse return wave signals, the point cloud data indicating a distance between the device and the target object.

2. The apparatus of claim 1, wherein, When generating the point cloud data of the target object based on the forward return wave signals and the reverse return wave signals, the point cloud generating module is specifically configured to: determine a position deviation of the target object based on the forward return wave signals and the reverse return wave signals, and generate first point cloud data based on the position deviation of the target object.

3. The apparatus of claim 1, wherein, When generating the point cloud data of the target object based on the forward return wave signals and the reverse return wave signals, the point cloud generating module is specifically configured to: determine a time difference based on the forward return wave signals and the reverse return wave signals, and generate first point cloud data based on the time difference.

4. The device of any one of claims 1-3, wherein, The device further comprises an image generating module. The image generating module is configured to generate a target image based on the return wave signals received by the return wave receiving module, the target image comprising the target object.

5. The device of any one of claims 1-4, wherein, The device further comprises an image generating module. The image generating module is configured to generate a forward scanning image based on the forward return wave signals, and generate a reverse scanning image based on the reverse return wave signals, the forward scanning image and the reverse scanning image both comprising the target object. The position deviation is determined according to a first position and a second position, the first position being a position of the target object in the forward scanning image, and the second position being a position of the target object in the reverse scanning image.

6. The apparatus of claim 2 or 3, wherein, The return wave receiving module is further configured to generate a depth signal corresponding to each return wave signal, the depth signal being used to indicate the distance between the device and the target object. When generating the point cloud data of the target object, the point cloud generating module is specifically configured to: generate second point cloud data based on the first point cloud data and at least one depth signal, the second point cloud data having a higher accuracy than the first point cloud data.

7. The apparatus of claim 5, wherein, The return wave receiving module is further configured to generate a gray scale signal corresponding to each return wave signal based on a signal strength of at least one return wave signal, the gray scale signal being used to indicate a light and dark degree of the scanned object. The forward scanning image is generated based on the gray scale signal corresponding to each forward return wave signal in at least one forward return wave signal, and the reverse scanning image is generated based on the gray scale signal corresponding to each reverse return wave signal in at least one reverse return wave signal.

8. The apparatus of claim 5, wherein, The echo receiving module is further configured to generate a depth signal corresponding to at least one echo signal, the depth signal being used to indicate a distance between the device and the target object; The forward scanning image is generated based on a depth signal corresponding to each of the at least one forward echo signal, and the backward scanning image is generated based on a depth signal corresponding to each of the at least one backward echo signal.

9. The device of any one of claims 5-8, wherein, The beam scanning module is further configured to generate a reference signal, the reference signal being used to indicate a scanning direction and / or a scanning position; The image generating module is specifically configured to: generate a pixel mapping relationship based on the reference signal, the pixel mapping relationship being used to indicate a corresponding echo signal of each pixel included in a frame of image, and a value of each pixel being determined based on the corresponding echo signal of the pixel; wherein the forward scanning image includes pixels corresponding to the forward echo signal in the frame of image, and the backward scanning image includes pixels corresponding to the backward echo signal in the frame of image.

10. The device of any one of claims 5-9, wherein, The position deviation of the target object is determined in the following manner: instance segmentation is performed on the forward scanning image and the backward scanning image to obtain a first target object instance in the forward scanning image and a second target object instance in the backward scanning image; The position deviation of the target object is a distance between a first position of the first target object instance in the forward scanning image and a second position of the second target object instance in the backward scanning image.

11. The apparatus of any one of claims 5-9, wherein, The position deviation of the target object is determined in the following manner: feature extraction is performed on the forward scanning image to obtain a feature corresponding to each pixel included in the forward scanning image, the feature including a feature used to indicate the target object; feature extraction is performed on the backward scanning image to obtain a feature corresponding to each pixel included in the backward scanning image, the feature including a feature used to indicate the target object; The position deviation of the target object includes a distance between a position of a pixel corresponding to a first feature in the forward scanning image and a position of a pixel corresponding to the first feature in the backward scanning image, the first feature being any feature attributed to the target object.

12. The apparatus of claim 4, wherein The target image is a frame of image determined based on the echo signal; or The target image is the forward scanning image, the forward scanning image being an image generated based on the forward echo signal in the frame of image; or The target image is the backward scanning image, the forward scanning image being an image generated based on the backward echo signal in the frame of image; or The target image is an image determined based on the forward scanning image and the backward scanning image.

13. A method of generating point cloud data, the method comprising: The method comprises: acquire echo information, the echo information comprising information of echo signals reflected by a target object scanned in a scanning direction, wherein the scanning direction comprises a first direction and a second direction, and the echo signals comprise forward echo signals and reverse echo signals, the forward echo signals being reflected by electromagnetic waves scanning the target object in the first direction, and the reverse echo signals being reflected by electromagnetic waves scanning the target object in the second direction; generate point cloud data of the target object based on the forward echo signals and the reverse echo signals, the point cloud data indicating a distance between the device and the target object.

14. The method of claim 13, wherein, The generating of the point cloud data of the target object based on the forward echo signals and the reverse echo signals comprises: determining a position deviation of the target object based on the forward echo signals and the reverse echo signals, and generating first point cloud data based on the position deviation of the target object.

15. The method of claim 13, wherein, The generating of the point cloud data of the target object based on the forward echo signals and the reverse echo signals comprises: determining a time difference based on the forward echo signals and the reverse echo signals, and generating first point cloud data based on the time difference.

16. The method according to any one of claims 13 to 15, wherein, The method further comprises: generating a target image based on the received echo signals, the target image comprising the target object.

17. The method of any one of claims 13-16, wherein, The method further comprises: generating a forward scanning image based on the forward echo signals, and generating a reverse scanning image based on the reverse echo signals, the forward scanning image and the reverse scanning image both comprising the target object; The position deviation of the target object is determined according to a first position and a second position, the first position being a position of the target object in the forward scanning image, and the second position being a position of the target object in the reverse scanning image.

18. The method of claim 14 or 15, wherein, The method further comprises: generating a depth signal corresponding to each echo signal, the depth signal being used to indicate a distance between the device and the target object; The generating of the point cloud data of the target object based on the forward echo signals and the reverse echo signals comprises: generating second point cloud data based on the first point cloud data and at least one depth signal, the second point cloud data having a higher accuracy than the first point cloud data.

19. The method of claim 17, wherein, The position deviation of the target object is determined in the following manner: performing instance segmentation on the forward scanning image and the reverse scanning image to obtain a first target object instance in the forward scanning image and a second target object instance in the reverse scanning image; The position deviation of the target object is a distance between a first position of the first target object instance in the forward scanning image and a second position of the second target object instance in the reverse scanning image.

20. The method of claim 17, wherein, The position deviation of the target object is determined in the following manner: performing feature extraction on the forward scanning image to obtain a feature corresponding to each pixel of a plurality of pixels included in the forward scanning image, the feature comprising a feature used to indicate the target object; performing feature extraction on the reverse scan image to obtain a feature corresponding to each pixel in the reverse scan image, the feature including a feature indicating the target object; The position deviation of the target object includes a distance between a position of a pixel corresponding to the first feature in the forward scan image and a position of a pixel corresponding to the first feature in the reverse scan image, the first feature being any one feature attributed to the target object.

21. The method of any one of claims 13-20, wherein, The target image is one frame of image determined based on the echo signal; or The target image is the forward scan image, the forward scan image being an image generated based on the forward echo signal in the one frame of image; or The target image is the reverse scan image, the forward scan image being an image generated based on the reverse echo signal in the one frame of image; or The target image is an image determined based on the forward scan image and the reverse scan image.

22. A vehicle characterized by The detection device according to any one of claims 1-12.

23. A camera, comprising: The detection device according to any one of claims 1-12.

24. A terminal device, comprising: The detection device according to any one of claims 1-12.

Images