Data processing method, data processing device, data acquisition device and vehicle

Abstract

The present disclosure relates to a data processing method, a data processing device, a data acquisition device and a vehicle, and relates to the field of vehicle technology, including: controlling a first acquisition device to send a synchronization signal to a second acquisition device at a first time point; the synchronization signal instructs the second acquisition device to acquire image data of a target object upon receiving the synchronization signal, and there is a communication delay between the first acquisition device and the second acquisition device; controlling the first acquisition device to acquire point cloud data of the target object at a second time point; the interval between the first time point and the second time point is equal to the communication delay. Using the data processing method proposed in the present disclosure, the first acquisition device and the second acquisition device can be controlled to acquire data simultaneously, so as to achieve alignment of point cloud data and image data.

Description

Technical Field

[0001] The present disclosure relates to the field of vehicle technology, and in particular to a data processing method, a data processing device, a data acquisition device and a vehicle. Background Art

[0002] At present, vehicles are equipped with a large number of sensors, such as radars and cameras. In order to accurately identify targets in the vehicle's surrounding environment, the point cloud data collected by the radar and the image data collected by the camera can be fused to obtain accurate object information of the target object.

[0003] Before fusing the point cloud data collected by the radar with the image data collected by the camera, the point cloud data and the image data need to be aligned. However, there are still differences between the aligned point cloud data and the image data, so that the object information of the target object cannot be accurately obtained. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the present disclosure provides a data processing method, a data processing device, a data acquisition device and a vehicle.

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided a data processing method, including:

[0006] Controlling the first acquisition device to send a synchronization signal to the second acquisition device at a first time point; the synchronization signal instructs the second acquisition device to acquire image data of the target object upon receiving the synchronization signal, and there is a communication delay time between the first acquisition device and the second acquisition device;

[0007] Control the first acquisition device to acquire point cloud data of the target object at a second time point; the interval between the first time point and the second time point is equal to the communication delay duration.

[0008] Optionally, controlling the first acquisition device to send a synchronization signal to the second acquisition device at a first time point includes:

[0009] When the position of the rotating mirror in the first acquisition device reaches the first position, determining that the first time point has been reached;

[0010] Control the first acquisition device to send the synchronization signal to the second acquisition device at the first time point.

[0011] Optionally, controlling the first acquisition device to acquire point cloud data of the target object at a second time point includes:

[0012] When the position of the rotating mirror in the first acquisition device reaches a second position, determining that the second time point has been reached;

[0013] Control the first acquisition device to acquire the point cloud data at the second time point.

[0014] Optionally, the method further comprises:

[0015] The first time point is determined according to the second time point and the communication delay duration.

[0016] Optionally, the arrival of the rotating mirror position in the first acquisition device at a first position indicates the arrival of the first time point, and the arrival of the rotating mirror position in the first acquisition device at a second position indicates the arrival of the second time point; and determining the first time point according to the second time point and the communication delay duration includes:

[0017] Obtaining a rotation angle of the rotating mirror according to the communication delay time and the rotation angular velocity of the rotating mirror;

[0018] The first position is determined according to the second position and the rotation angle.

[0019] Optionally, the method further comprises:

[0020] Obtaining a turning speed of the rotating mirror according to a frequency at which the first acquisition device acquires the point cloud data and the number of mirror surfaces of the rotating mirror;

[0021] The rotation angular velocity is obtained according to the steering speed.

[0022] Optionally, controlling the first acquisition device to send a synchronization signal to the second acquisition device at a first time point includes:

[0023] The first acquisition device is controlled to forward the synchronization signal to the second acquisition device through a controller at the first time point.

[0024] Optionally, the method further comprises:

[0025] The communication delay duration is obtained according to a first data transmission duration between the first acquisition device and the controller, a second data transmission duration between the second acquisition device and the controller, and a third data transmission duration for data transmitted within the controller.

[0026] According to a second aspect of an embodiment of the present disclosure, there is provided a data processing device, including:

[0027] a synchronization module, configured to control the first acquisition device to send a synchronization signal to the second acquisition device at a first time point; the synchronization signal instructs the second acquisition device to acquire image data of the target object upon receiving the synchronization signal, and there is a communication delay time between the first acquisition device and the second acquisition device;

[0028] The acquisition module is configured to control the first acquisition device to acquire point cloud data of the target object at a second time point; the interval between the first time point and the second time point is equal to the communication delay time.

[0029] According to a third aspect of an embodiment of the present disclosure, a chip is provided, comprising a processing unit and an interface circuit, wherein the processing unit obtains program instructions through the interface circuit, and the program instructions are executed by the processing unit, and the processing unit is used to execute the steps of the data processing method provided in the first aspect of an embodiment of the present disclosure.

[0030] According to a fourth aspect of an embodiment of the present disclosure, a data acquisition device is provided, on which a chip as provided in the third aspect of an embodiment of the present disclosure is configured.

[0031] According to a fifth aspect of an embodiment of the present disclosure, a vehicle is provided, on which is configured a data acquisition device as provided in the fourth aspect of an embodiment of the present disclosure.

[0032] According to a sixth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the program instructions are executed by a processor, the steps of the data processing method provided in the first aspect of the present disclosure are implemented.

[0033] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0034] Control sending a synchronization signal to the second acquisition device at a first time point, and control the first acquisition device to collect point cloud data at a second time point, and the interval between the first time point and the second time point is equal to the communication delay duration.

[0035] Under this scheme, since there is a communication delay between the second acquisition device and the first acquisition device, after the first acquisition device sends a synchronization signal at the first time point, the synchronization signal will reach the second acquisition device after the communication delay, and the second acquisition device will also receive the synchronization signal at the second time point after the delay, and respond to the synchronization signal to collect image data at the second time point. At the same time, the first acquisition device will also collect point cloud data at the second time point, so that the first acquisition device and the second acquisition device both obtain data at the same time point, so that the point cloud data collected by the first acquisition device and the image data collected by the second acquisition device can be aligned, and the data of the target object obtained after alignment and fusion is also accurate.

[0036] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0038] Figure 1 The invention is a flowchart of a data processing method according to an exemplary embodiment.

[0039] Figure 2 The diagram is a schematic diagram showing misalignment of point cloud data and image data according to an exemplary embodiment.

[0040] Figure 3 The figure is a schematic diagram showing the alignment of point cloud data and image data according to an exemplary embodiment.

[0041] Figure 4 is a schematic diagram of a rotating mirror module according to an exemplary embodiment.

[0042] Figure 5 is a schematic diagram of a rotating mirror module according to an exemplary embodiment.

[0043] Figure 6 is a schematic diagram of a pulse signal according to an exemplary embodiment.

[0044] Figure 7 The diagram is a schematic diagram showing the interaction among a first collection device, a domain controller and a second collection device according to an exemplary embodiment.

[0045] Figure 8 It is a logical schematic diagram of a data processing method according to an exemplary embodiment.

[0046] Fig. 9It is a schematic diagram showing a first data transmission duration, a second data transmission duration and a third data transmission duration according to an exemplary embodiment.

[0047] Fig.10 It is a block diagram of a data processing device according to an exemplary embodiment.

[0048] Fig.11 is a block diagram of a vehicle according to an exemplary embodiment.

[0049] Fig.12 is a block diagram of a chip according to an exemplary embodiment. DETAILED DESCRIPTION

[0050] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0051] It should be noted that all actions of acquiring signals, information or data in the present disclosure are carried out in compliance with the relevant data protection laws and policies of the country where the device is located and with the authorization given by the owner of the corresponding device.

[0052] Figure 1 is a flow chart of a data processing method according to an exemplary embodiment. Figure 1 As shown, the data processing method can be applied to the first chip, including the following steps.

[0053] In step S10, the first acquisition device is controlled to send a synchronization signal to the second acquisition device at a first time point.

[0054] The first acquisition device may be a device for acquiring point cloud data of target objects around the vehicle, for example, a radar device, which may be any one of laser radar, millimeter wave radar and other radar devices. The first acquisition device measures the distance and shape of the target object by emitting light and receiving the light reflected by the target object. In order to scan a preset area around the vehicle, the first acquisition device is usually provided with a rotating mirror, which changes the emission direction of the light through the rotating mirror, thereby realizing scanning of the preset area and obtaining the shape of the target object in the preset area and the distance between the target object and the first acquisition device.

[0055] The second acquisition device is a device for acquiring image data of target objects around the vehicle, and may be, for example, at least one of a surround view camera, a side view camera, and a front view camera, and these cameras may be CMOS (Complementary Metal Oxide Semiconductor) cameras. The second acquisition device captures images formed by external light through a photosensitive element such as a CMOS sensor. When external light shines on the sensor, the second acquisition device automatically adjusts the exposure time according to the light intensity to ensure the brightness and clarity of the image. Once the exposure is completed, the second acquisition device can output image data.

[0056] The synchronization signal is a signal used to trigger the second acquisition device to acquire image data. The synchronization signal instructs the second acquisition device to acquire image data of the target object when receiving the synchronization signal.

[0057] There is a communication delay time between the first acquisition device and the second acquisition device, so after the first acquisition device sends out a synchronization signal, the synchronization signal will arrive at the second acquisition device after being delayed by the communication delay time.

[0058] For example, taking the communication delay time as 50us, after the first acquisition device sends out the synchronization signal, the second acquisition device will delay 50us to receive the synchronization signal, so the second acquisition device will collect image data at the time point of 50us after the first acquisition device sends out the synchronization signal.

[0059] In step S20, the first acquisition device is controlled to acquire point cloud data of the target object at a second time point.

[0060] The interval duration between the second time point and the first time point is equal to the communication delay duration, so that the time point when the first acquisition device collects point cloud data can be the same as the time point when the second acquisition device collects image data, the time point when the first acquisition device and the second acquisition device collect the same target object is both at the second time point, and the exposure time point of the first acquisition device and the second acquisition device is both at the second time point.

[0061] Taking the communication delay time of 50us as an example, at the second time point of 50us after the first acquisition device is controlled to send a synchronization signal at the first time point, the first acquisition device will be controlled to collect point cloud data of the target object.

[0062] In some scenarios, taking the first time point as the 1us and the second time point as the 51us as an example, the interval duration of 50us between the first time point and the second time point is equal to the communication delay duration of 50us. After the first acquisition device is controlled to send a synchronization signal to the second acquisition device at the 1us, it reaches the second acquisition device after a communication delay duration of 50us, and the second acquisition device acquires the image data of the target object at the 51us; and the first acquisition device is controlled to acquire the point cloud data of the target object at the second time point of 51us, so that the time points when the first acquisition device and the second acquisition device acquire the image data and point cloud data of the same target object are aligned, and there will be no deviation.

[0063] It can be understood that the second time point when the first acquisition device collects point cloud data of the target object is the time point when the first acquisition device emits light to scan the target object, and the second time point when the second acquisition device collects image data of the target object is the time point when the second acquisition device is exposed. Therefore, aligning the time point when the first acquisition device collects the point cloud data of the target object with the time point when the second acquisition device collects the image data of the target object can also be understood as aligning the time point when the first acquisition device emits light with the time point when the second acquisition device is exposed.

[0064] In the related technology, the first acquisition device and the second acquisition device have different data acquisition frequencies. For example, the first acquisition device can use a frequency of 10Hz~20Hz to acquire point cloud data of the target object and attach a acquisition timestamp to each frame of point cloud data acquired, while the second acquisition device can use a frequency of 30Hz to acquire image data of the target object and attach a acquisition timestamp to each frame of image data acquired. Then, point cloud data similar to the multi-frame point cloud data acquired by the first acquisition device and the multi-frame image data acquired by the second acquisition device are selected to align and fuse with the image data. On the one hand, the point cloud data and image data aligned by this scheme have errors. They are not the data acquired by the first acquisition device and the second acquisition device at the same time. Especially when the vehicle speed is high, the error will be greater. Please refer to Figure 2 As shown in the figure, taking the target object as a triangle sign as an example, there is an error between the image data of the triangle sign on the left and the point cloud data on the right. , the two are not aligned; secondly, for each frame of point cloud data, it is necessary to calculate the image data close to the time point of the point cloud data, and the amount of image data and point cloud data is relatively large, which means that calculation operations will be performed frequently to calculate the image data and point cloud data close in time, which will take up a lot of computing resources.

[0065] For example, the acquisition frequency of the first acquisition device is 10Hz, and it collects point cloud data 10 times in 1 second, and collects point cloud data at the 100ms, 200ms, 300ms, 400ms... and other time points respectively; the acquisition frequency of the second acquisition device is 30Hz, and it collects image data 30 times in 1 second, and collects image data at the 30ms, 60ms, 90ms, 120ms, 150ms... and other time points respectively. Since the point cloud data at the 100th ms and the image data at the 90th ms are close in time, the point cloud data at the 100th ms and the image data at the 90th ms will be aligned and fused, but the point cloud data at the 100th ms and the image data at the 90th ms are still not data collected at the same time point; and when the vehicle speed is high and the moving speed of the target object is high, the image data collected by the second acquisition device at the 90th ms may be target A, and the point cloud data collected by the second acquisition device at the 100th ms may be target B. The data of the target object obtained by fusing the image data and point cloud data of two objects that are not the same target is also wrong.

[0066] In the embodiment of the present disclosure, on the first aspect, the problem will be solved from the source. The first acquisition device is controlled to acquire point cloud data at a second time point at which the communication delay duration is after the first time point, and a synchronization signal is sent to the second acquisition device. Since there is a communication delay duration between the second acquisition device and the first acquisition device, the second acquisition device will also receive the synchronization signal at the second time point, and respond to the synchronization signal to acquire image data at the second time point, so that the time point at which the first acquisition device acquires point cloud data is the same as the time point at which the second acquisition device acquires image data. The two acquire image data and point cloud data at the same time point. Even if the vehicle speed is high or the target object moves at a high speed, the first acquisition device and the second acquisition device acquire data at the same time point, so that the point cloud data and the image data can be aligned, and the data of the target object obtained after fusion is also accurate. Please refer to Figure 3 As shown in the figure, taking the target object as a triangle sign as an example, the image data and the point cloud data are strictly aligned, which achieves pixel-level alignment. Pixel-level alignment means that for each point cloud data, the corresponding pixel position can be found in the image. Secondly, it does not need to filter the image data close to the time point of the point cloud data, which reduces the amount of calculation and reduces the occupation of computing resources.

[0067] Figure 4 and Figure 5 It is an exemplary scheme for determining the arrival of the first time point proposed according to an exemplary embodiment. The embodiment of the present disclosure determines the arrival of the first time point by the rotating mirror position reaching the first position, and accumulates the time points by counting the mechanical turntable, which includes the following steps.

[0068] (1) When the position of the rotating mirror in the first acquisition device reaches a first position, determining that the first time point has been reached.

[0069] The first acquisition device includes not only a rotating mirror but also a scale plate, which is also called a coding plate. Figure 4 and Figure 5 As shown, the scale plate and the rotating mirror are integrally arranged, and a driving motor is arranged below the scale plate, and the driving motor drives the scale plate to rotate, thereby driving the rotating mirror on the driving plate to rotate accordingly. The scale plate is provided with multiple scales, and the first position is the first preset scale on the scale plate. When the rotating mirror position rotates to the first preset scale, it means that the first time point has been reached, and at this time, a synchronization signal can be sent to the second acquisition device.

[0070] The position of the rotating mirror can be obtained by any of the following methods A and B.

[0071] In mode A, a reading acquisition device may be arranged around the first acquisition device, the reading acquisition device is used to read the scale on the dial, the reading acquisition device sends a pulse signal to the dial, the dial reflects the pulse signal to the reading acquisition device, and the reading acquisition device determines the position of the rotating mirror according to the number of the returned pulse signals. Among them, the specified number of pulse signals is equal to one scale on the dial, and the position of the rotating mirror differs from the first position by a first preset number of scales. Since the position of the rotating mirror can be represented by the zero position on the dial, the difference between the position of the rotating mirror and the first position by the first preset number of scales can be the difference between the zero position on the dial and the first position by the first preset number of scales.

[0072] When the zero position on the scale plate rotates by a first preset number of scales, it is determined that the rotating mirror rotates by the first preset number of scales, thereby determining that the position of the rotating mirror reaches the first position.

[0073] For example, if 100 pulse signals equal one scale, and the difference between the first position and the zero position is 5 first preset scales, the reading acquisition device will identify the zero position on the dial by the width of the pulse signal. Figure 6 As shown, when there is a pulse signal with a larger width among the pulse signals acquired by the reading acquisition device, it means that the reading acquisition device has acquired a pulse signal at the zero position on the dial. At this time, the reading acquisition device can start counting the pulse signals with a narrower width. If 300 pulse signals are identified, it means that the dial has rotated 3 scales, the zero position on the dial has rotated 3 scales, and the dial also drives the rotating mirror to rotate 3 scales. When the reading acquisition device identifies 500 pulse signals, it means that the dial has rotated 5 scales, the zero position on the dial has rotated 5 scales, and it can be determined that the rotating mirror position has rotated to the first position.

[0074] Method B, a Hall sensor can be set under the driving motor that drives the dial to rotate, and the Hall sensor can be used to sense the zero position on the dial to obtain the rotating mirror position. When the rotating mirror position reaches the first position, it is determined that it has reached the first time point, and a synchronization signal can be issued.

[0075] (2) Controlling the first acquisition device to send the synchronization signal to the second acquisition device at the first time point.

[0076] Through the above technical solution, it is possible to determine whether the rotating mirror position has reached the first position by reading the number of scales that the rotating mirror position in the first acquisition device has rotated. When the rotating mirror position has rotated through a first preset number of scales, it is determined that the rotating mirror position is at the first position, indicating that the first time point has been reached, so that a synchronization signal can be issued. In the case where the first time point cannot be clearly obtained, the arrival of the first time point can be represented by the arrival of the rotating mirror position at the first position, and a synchronization signal is issued when the rotating mirror position reaches the first position.

[0077] Figure 4 and Figure 5 An exemplary scheme for determining arrival at a second time point is proposed according to an exemplary embodiment, and includes the following steps.

[0078] (1) When the position of the rotating mirror in the first acquisition device reaches the second position, determining that the second time point has been reached.

[0079] The second position is the second preset scale on the dial. When the mirror position is rotated to the second preset scale, it means that the second time point has been reached. At this time, the first acquisition device can be controlled to collect point cloud data.

[0080] The mirror position differs from the second position by a second preset number of scales. Since the mirror position can be represented by the zero position on the dial, the mirror position differs from the second position by a second preset number of scales can be the zero position on the dial and the second position differ from the second preset number of scales, and the second preset number is greater than the first preset number.

[0081] When the zero position on the scale plate rotates by a second preset number of scales, it is determined that the rotating mirror position rotates by a second preset number of scales, and it is determined that the rotating mirror position reaches the second position.

[0082] For example, taking 100 pulse signals equal to 1 scale, and the difference between the second position and the zero position is 10 second preset number of scales as an example, the reading acquisition device will identify the zero position on the dial by the width of the pulse signal. When there is a pulse signal with a larger width among the pulse signals acquired by the reading acquisition device, it means that the reading acquisition device has acquired the pulse signal of the zero position on the dial. At this time, the reading acquisition device can start counting the pulse signals with a narrower width. If 300 pulse signals are identified, it means that the dial has rotated 3 scales, the zero position on the dial has rotated 3 scales, and the dial also drives the rotating mirror to rotate 3 scales. When the reading acquisition device identifies 1000 pulse signals, it means that the dial has rotated 10 scales, the zero position on the dial has rotated 10 scales, and it can be determined that the rotating mirror position has rotated to the second position.

[0083] Furthermore, the duration that the rotating mirror rotates from the first position to the second position is the interval between the first time point and the second time point, and the interval is equal to the communication delay.

[0084] (2) Controlling the first acquisition device to acquire the point cloud data at the second time point.

[0085] Through the above technical solution, it is possible to determine whether the rotating mirror position has reached the second position by reading the number of scales that the rotating mirror position in the first acquisition device has rotated. When the rotating mirror position has rotated through the second preset number of scales, it is determined that the rotating mirror position is at the second position, indicating that the second time point has been reached, so that the first acquisition device can be controlled to collect point cloud data. In the case where the second time point cannot be clearly obtained, it can represent that the second time point has been reached by the rotating mirror position reaching the second position, and a synchronization signal is sent when the rotating mirror position reaches the second position.

[0086] Figure 4 and Figure 5 It is an exemplary scheme for determining the first time point proposed according to an exemplary embodiment, which includes: obtaining the first time point according to the second time point and the communication delay duration, which includes the following two exemplary schemes.

[0087] In the first solution, the first time point can be represented by the rotating mirror position in the first acquisition device reaching the first position, and the second time point can be represented by the rotating mirror position in the second acquisition device reaching the second position. Under this condition, the first time point is determined by the following steps.

[0088] Step A1: Obtain the turning speed of the rotating mirror according to the frequency at which the first acquisition device acquires the point cloud data and the number of mirror surfaces of the rotating mirror.

[0089] The number of mirror surfaces of the rotating mirror can be configured according to the configuration requirements of the first acquisition device. If the number of mirror surfaces of the rotating mirror is 3, it means that the rotating mirror is a prism, and the mirror surfaces of the three sides thereof all have a reflective effect.

[0090] The frequency of collecting point cloud data by the first collection device can be divided by the number of mirror surfaces of the reflector, and then multiplied by 60 to obtain the turning speed of the rotating mirror, which is also the turning speed of the driving motor. The turning speed of the rotating mirror can be calculated by the following calculation formula:

[0091] (1)

[0092] In formula (1), v is the turning speed of the rotating mirror; f is the frequency at which the first acquisition device acquires point cloud data; and n is the number of mirror surfaces of the reflector.

[0093] Step A2: Obtain the rotation angular velocity according to the steering speed.

[0094] The angular velocity can be obtained by multiplying the steering speed by 360 and then dividing by 60. 360 indicates the number of scales on the dial, which is calculated by the following formula:

[0095] (2)

[0096] In formula (2), is the angular velocity of rotation; v is the turning speed of the mirror.

[0097] Step A3: Obtaining the rotation angle of the rotating mirror according to the communication delay time and the rotation angular velocity of the rotating mirror.

[0098] The communication delay time can be multiplied by the rotation angular velocity to obtain the rotation angle of the rotating mirror caused by the communication delay time. The rotation angle is also the angle between the first position and the second position. The calculation formula is as follows:

[0099] (3)

[0100] In formula (3), is the rotation angle of the mirror caused by the communication delay; is the angular velocity of rotation; is the communication delay duration.

[0101] Step A4: Obtain the first position according to the second position and the rotation angle.

[0102] The first position can be obtained by subtracting the rotation angle from the second position.

[0103] Since the second position is the second preset angle on the dial and the first position is the first preset angle on the dial, the first preset angle can be obtained by subtracting the rotation angle caused by the communication delay time from the second preset angle.

[0104] It can be understood that the various time points at which the first acquisition device collects point cloud data of the target object are not controlled, but the various time points at which the second acquisition device collects image data of the target object can be controlled. Therefore, the various second time points at which the first acquisition device collects point cloud data can be obtained, and then the various first time points for sending synchronization signals can be reversely calculated through the various second time points and the communication delay duration, and then the second acquisition device can be controlled to collect image data at each second time point of the communication delay duration after each first time point, so as to change the time point of the image data collected by the second acquisition device to the second time point without changing the second time point at which the first acquisition device collects point cloud data, so that the first acquisition device and the second acquisition device can synchronously collect data at each second time point, and the frequency of the second acquisition device collecting image data is changed to be consistent with the frequency of the first acquisition device collecting point cloud data, and the two collect data at the same frequency.

[0105] Moreover, different second time points correspond to different first time points. For example, when the second time points at which the first acquisition device acquires point cloud data are 100ms, 200ms, 300ms, 400ms, and 500ms, respectively, if the communication delay is 10ms, then the first time points at which the corresponding first acquisition device sends a synchronization signal to the second acquisition device are 90ms, 190ms, 290ms, 390ms, and 490ms, respectively. Since there is a 10ms communication delay between the first acquisition device and the second acquisition device, the second acquisition device receives the synchronization signal sent by the first acquisition device at 100ms, 200ms, 300ms, 400ms, and 500ms, respectively, and then acquires image data at 100ms, 200ms, 300ms, 400ms, and 500ms, respectively.

[0106] Through the first solution, when the first time point and the second time point cannot be obtained, the rotation angle of the mirror position after the communication delay period can be determined first, and the rotation angle can be subtracted from the second position to determine the first position that can represent the first time point, so that the synchronization signal can be accurately sent at the first time point.

[0107] In a second solution, the first time point may be obtained by subtracting the communication delay duration from the second time point.

[0108] Through the second solution, when the first time point and the second time point can be obtained, the communication delay duration can be subtracted from the second time point to obtain the first time point when the first acquisition device sends a synchronization signal.

[0109] Figure 7 and Figure 8 An exemplary solution is shown in which a first acquisition device sends a synchronization signal to a second acquisition device at a first time point, which includes: controlling the first acquisition device to forward the synchronization signal to the second acquisition device through a first chip at the first time point. The first chip may be a first chip configured in the first acquisition device.

[0110] For example, see Figure 8 As shown, after the process starts, the first chip obtains the position of the rotating mirror, and determines whether it has reached the first time point through the rotating mirror position. After reaching the first time point, it sends a synchronization signal to the second chip, and the second chip captures the synchronization signal; the second chip then sends a synchronization signal to the second acquisition device, and the second acquisition device collects image data after receiving the synchronization signal. At the same time, the first acquisition device collects point cloud data, and then fuses the synchronously triggered image data and point cloud data, thereby ending the process. Of course, before reaching the first time point, the first chip will continue to obtain the position of the rotating mirror.

[0111] See also Figure 7 As shown, the data processing method proposed in the present disclosure is applied in the first chip of the first acquisition device. Figure 7 As shown, it includes a first acquisition device (Lidar), a domain controller (ADD) and a second acquisition device.

[0112] Among them, the first acquisition device includes a light source (VCSEL IIIumination), a receiver, a first chip (FPGA or CPLD), a memory (ROM), a rotating mirror module (Polygon Mirror), a serializer, a clock (clock), a power supply (power) and a microcontroller (MCU).

[0113] The light source may be a laser light source for emitting laser light, and the laser light is transmitted to the rotating mirror as incident light, and the rotating mirror emits reflected light for scanning point cloud data of the target object.

[0114] The receiver is used to receive the synchronization signal.

[0115] The memory is used to store signals transmitted by the receiver, such as synchronization signals.

[0116] The microcontroller is used to control the first acquisition device as a whole, and the microcontroller communicates with the receiver, the first chip and the serializer through a serial communication protocol (Inter-Integrated Circuit, IIC).

[0117] The rotating mirror module is used to receive incident light and generate reflected light in different directions by rotating the rotating mirror, so as to scan targets in different preset areas. The rotating mirror module includes a reading device, a rotating mirror, a driving motor, a dial, etc. The rotating mirror module is used to read the position of the rotating mirror, which can also be understood as reading the number of scales rotated by the zero position on the dial. The rotating mirror module can also control the speed of the driving motor, and can also perform real-time encoding of the horizontal scanning angle of the first acquisition device, and can also set multiple second time points for the first acquisition device to emit light, that is, set each second position where the first acquisition device emits light.

[0118] The first chip is used to generate a synchronization signal. The first chip can detect the first time point and the second time point, that is, determine whether the rotating mirror position reaches the first position or the second position. The first chip can also send a synchronization signal to the serializer through the first interface (Mobile Industry Processor Interface, MIPI) when the rotating mirror position reaches the first position; when the rotating mirror position reaches the second position, the first acquisition device is controlled to emit laser light; the first chip can also control the timing relationship between the laser light emitted by the first acquisition device and the light returned by the first acquisition device from the target object. Data is transmitted between the first chip and the receiver through the first interface, and data is also transmitted between the first chip and the serializer through the first interface (Mobile Industry Processor Interface, MIPI).

[0119] The serializer is used to encode the synchronization signal sent by the first chip, generate a synchronization signal that complies with the serial link (GigabitMultimedia Serial Link, GMSL), and then transmit the synchronization signal to the domain controller through the forward channel (tunneling) of the serial link.

[0120] See also Figure 7 As shown, the first chip generates a synchronization signal (F_Sync), and sends the synchronization signal to the receiver and the serializer through the first interface, and the first chip also transmits the synchronization signal to the light source at a second time point to trigger the light source to emit a laser beam to scan the target object; after the serializer receives the synchronization signal, it sends the synchronization signal to the domain controller. It can be understood that the synchronization signal generated by the first chip in the embodiment of the present disclosure can also be called a trigger signal (Trigger), which is used to trigger the light source to emit a laser beam, or trigger the second acquisition device to collect image data.

[0121] The domain controller is used to process the synchronization signal and forward the synchronization signal to the second acquisition device. The domain controller includes a deserializer and a second chip (System on Chip, SOC), and the second chip includes a second interface (CameraSerial Interface, CSI), a chip memory interrupt (SOC MEM interrupt), a general input and output interrupt (SOCGPIO interrupt) and a general input and output controller (SOC GPIO Controller).

[0122] The deserializer is used to receive the synchronization signal that complies with the multimedia serial link output by the serializer in the first acquisition device, and parse the synchronization signal that complies with the multimedia serial link, and transmit the parsed synchronization signal to the second interface; the second interface transmits the parsed synchronization signal to the universal input and output interrupter and the universal input and output controller, the universal input and output controller captures the synchronization signal to trigger the universal input and output interrupter, the interrupt processing function in the universal input and output interrupter generates a corresponding synchronization signal to trigger the second acquisition device, and the synchronization signal that triggers the second acquisition device to collect image data is transmitted to the second acquisition device through the deserializer.

[0123] Among them, the second acquisition device communicates with the deserializer in the domain controller through a serial link. The second acquisition device receives the synchronization signal sent by the deserializer in the domain controller through the serial link, and collects the image data of the target object in response to the synchronization signal. The second acquisition device includes a camera receiver (CMOS receiver) and a serializer. The camera receiver and the serializer transmit data through the first interface. The serializer receives the synchronization signal sent by the deserializer in the domain controller, parses the synchronization signal, and enables the sensor of the second acquisition device to expose and collect the image data of the target object based on the synchronization signal. In addition, the serializer in the second acquisition device can also transmit the collected image data to the domain controller through a serial link, and the domain controller performs fusion processing on the image data and the point cloud data.

[0124] Among them, see Figure 7 As shown, the deserializer is provided with a first pin (GPIO1) and a second pin (GPIO2), and the second chip is also provided with a first pin (GPIO1) and a second pin (GPIO2). The first pin in the deserializer transmits the data received by the deserializer to the first pin of the second chip, and the second pin in the second chip can also transmit the data processed by the second chip to the second pin of the deserializer.

[0125] In the related technology, as the safety of autonomous driving improves, the resolution of images collected by the second acquisition device gradually increases, for example, reaching 8 million pixels, and the equivalent line bundle of the first acquisition device also gradually increases, for example, from hundreds of lines to thousands of lines. In this way, the bandwidth for data transmission between the first acquisition device and the domain controller is not sufficient to support the rapid transmission of point cloud data, and the bandwidth for data transmission between the second acquisition device and the domain controller is not sufficient to support the rapid transmission of image data.

[0126] Through the above technical solution, by using GMSL serial link communication between the first acquisition device and the domain controller, and using GMSL serial link communication between the second acquisition device and the domain controller, since the GMSL interface is a gigabit multimedia serial link, it can support a data transmission speed of up to 12Gbps, so the first acquisition device can quickly transmit point cloud data to the domain controller through the GMSL serial link, and the second acquisition device can also quickly transmit point cloud data to the domain controller through the GMSL serial link, and the data transmission speed is increased.

[0127] Moreover, by matching the serializer with the deserializer, transparent transmission of the synchronization signal can be achieved, which can ensure that when the synchronization signal is transmitted between the first acquisition device and the domain controller, and when it is transmitted between the domain controller and the second acquisition device, no other processing or modification is performed on the synchronization data, thereby largely ensuring the originality and integrity of the synchronization signal.

[0128] Figure 7 and Fig. 9 This is an exemplary scheme for obtaining a communication delay duration proposed according to an exemplary embodiment, which includes: obtaining the communication delay duration according to a first data transmission duration between the first acquisition device and the controller, a second data transmission duration between the second acquisition device and the controller, and a third data transmission duration for data transmitted within the controller.

[0129] The controller can be Figure 7 and Fig. 9 The domain controller is shown.

[0130] The communication delay duration is equal to the sum of the first data transmission duration, the second data transmission duration and the third data transmission duration.

[0131] The first data transmission duration may be a data transmission duration between a serializer in the first acquisition device and a deserializer in the controller.

[0132] The second data transmission duration may be a data transmission duration between a serializer in the second acquisition device and a deserializer in the controller.

[0133] The third data transmission time is the transmission time between the deserializer in the controller, the interface in the controller, the general input and output controller, and the general input and output interrupter.

[0134] See also Figure 7 and Fig. 9 As shown, Fig. 9 There are two links connected to the second chip, the upper link is the link for communication between the domain controller and the first acquisition device, and the lower link is the communication link between the domain controller and the second acquisition device. The first chip in the first acquisition device transmits the synchronization signal to the serializer in the first acquisition device, and the serializer in the first acquisition device then transmits the synchronization signal to the deserializer in the domain controller. Then, the serializer in the first acquisition device transmits the synchronization signal to the deserializer in the controller, which requires the first data transmission time; the deserializer in the controller transmits the synchronization signal through the interface, general input and output controller, general input and output interrupter, etc. in the controller and then returns to the deserializer in the controller, thus generating a third data transmission time; the deserializer in the controller then transmits the synchronization signal to the serializer in the second acquisition device, then the deserializer in the controller requires the second data transmission time to transmit the synchronization signal to the serializer in the second acquisition device.

[0135] It can be seen that transmitting the synchronization signal from the first acquisition device to the second acquisition device requires at least the communication delay caused by the first data transmission duration, the second data transmission duration, and the third data transmission duration. Since the first data transmission duration between the first acquisition device and the domain controller is the communication duration between the serializer and the decoder, and the second data transmission duration between the second acquisition device and the domain controller is still the communication duration between the serializer and the decoder, when the executor of the transmitted data remains unchanged and the transmission links are all serial links, the first data transmission duration is equal to the second data transmission duration. Finally, the calculation formula for the communication delay duration is obtained based on the first data transmission duration, the second data transmission duration, and the third data transmission duration as follows:

[0136] (4)

[0137] In the above formula (4), is the communication delay duration; is the first data transmission duration or the second data transmission duration; is the third data transmission duration.

[0138] Through the above technical scheme, the communication delay duration between the first acquisition device and the second acquisition device can be calculated, and then after obtaining the second time point when the first acquisition device collects point cloud data, the first time point for sending the synchronization signal can be reversely calculated based on the communication delay duration. In this way, after the first acquisition station device sends the synchronization signal at the first time point, the synchronization signal reaches the second acquisition device after the communication delay duration, which is also the second time point. Only then can the first acquisition device and the second acquisition device collect data synchronously.

[0139] Fig.10 is a block diagram of a data processing device according to an exemplary embodiment. Fig.10 The data processing device 1000 includes: a synchronization module 1010 and a collection module 1020 .

[0140] The synchronization module 1010 is configured to control the first acquisition device to send a synchronization signal to the second acquisition device at a first time point; the synchronization signal instructs the second acquisition device to acquire image data of the target object upon receiving the synchronization signal, and there is a communication delay time between the first acquisition device and the second acquisition device;

[0141] The acquisition module 1020 is configured to control the first acquisition device to acquire the point cloud data of the target object at a second time point; the interval between the first time point and the second time point is equal to the communication delay time.

[0142] Optionally, the synchronization module 1010 includes:

[0143] A first identification submodule is configured to determine that the first time point has been reached when the rotating mirror position in the first acquisition device reaches a first position;

[0144] The synchronization submodule is configured to control the first acquisition device to send the synchronization signal to the second acquisition device at the first time point.

[0145] Optionally, the acquisition module 1020 includes:

[0146] A second identification submodule is configured to determine that the second time point has been reached when the rotating mirror position in the first acquisition device reaches a second position;

[0147] The acquisition submodule is configured to control the first acquisition device to acquire the point cloud data at the second time point.

[0148] Optionally, the data processing device 1000 includes:

[0149] The first time point module is configured to determine the first time point according to the second time point and the communication delay duration.

[0150] Optionally, when the rotating mirror position in the first acquisition device reaches a first position, it indicates that the first time point has been reached; when the rotating mirror position in the first acquisition device reaches a second position, it indicates that the second time point has been reached; the first time point module includes:

[0151] A rotation angle submodule, configured to obtain the rotation angle of the rotating mirror according to the communication delay time and the rotation angular velocity of the rotating mirror;

[0152] The first position submodule is configured to determine the first position according to the second position and the rotation angle.

[0153] Optionally, the data processing device 1000 includes:

[0154] A special speed module is configured to obtain a turning speed of the rotating mirror according to a frequency at which the first acquisition device acquires the point cloud data and the number of mirror surfaces of the rotating mirror;

[0155] The rotation angular velocity module is configured to obtain the rotation angular velocity according to the steering speed.

[0156] Optionally, the synchronization module 1010 is further configured to control the first acquisition device to forward the synchronization signal to the second acquisition device through a controller at the first time point.

[0157] Optionally, the data processing device 1000 includes:

[0158] The communication delay duration module is configured to obtain the communication delay duration according to the first data transmission duration between the first acquisition device and the controller, the second data transmission duration between the second acquisition device and the controller, and the third data transmission duration of the data transmitted within the controller.

[0159] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0160] The present disclosure also provides a computer-readable storage medium on which computer program instructions are stored. When the program instructions are executed by a processor, the steps of the data processing method provided by the present disclosure are implemented.

[0161] Based on the same inventive concept, the present disclosure further proposes a data acquisition device according to an exemplary embodiment. The data acquisition device is used to acquire point cloud data of a target object. The data acquisition device is configured with the above-mentioned first chip.

[0162] Fig.111 is a block diagram of a vehicle 1100 according to an exemplary embodiment. For example, the vehicle 1100 may be a hybrid vehicle, a non-hybrid vehicle, an electric vehicle, a fuel cell vehicle, or other types of vehicles. The vehicle 1100 may be an autonomous vehicle, a semi-autonomous vehicle, or a non-autonomous vehicle.

[0163] Reference Fig.11 , the vehicle 1100 may include various subsystems, such as an infotainment system 1110, a perception system 1120, a decision control system 1130, a drive system 1140, and a computing platform 1150. The vehicle 1100 may also include more or fewer subsystems, and each subsystem may include multiple components. In addition, each subsystem and each component of the vehicle 1100 may be interconnected by wire or wireless means.

[0164] In some embodiments, the infotainment system 1110 may include a communication system, an entertainment system, and a navigation system, etc.

[0165] The perception system 1120 may include several sensors for sensing information about the environment around the vehicle 1100. For example, the perception system 1120 may include a global positioning system (the global positioning system may be a GPS system, or a Beidou system or other positioning systems), an inertial measurement unit (IMU), a laser radar, a millimeter wave radar, an ultrasonic radar, and a camera.

[0166] The decision control system 1130 may include a computing system, a vehicle controller, a steering system, a throttle, and a braking system.

[0167] The drive system 1140 may include components that provide powered motion for the vehicle 1100. In one embodiment, the drive system 1140 may include an engine, an energy source, a transmission system, and wheels. The engine may be one or a combination of an internal combustion engine, an electric motor, and an air compression engine. The engine is capable of converting energy provided by the energy source into mechanical energy.

[0168] Some or all functions of the vehicle 1100 are controlled by a computing platform 1150. The computing platform 1150 may include at least one first processor 1151 and a memory 1152, and the first processor 1151 may execute instructions 1153 stored in the memory 1152.

[0169] The first processor 1151 may be any conventional processor, such as a commercially available CPU. The processor may also include a graphics processor (Graphic Process Unit, GPU), a field programmable gate array (Field Programmable Gate Array, FPGA), a system on chip (System on Chip, SOC), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC) or a combination thereof.

[0170] The memory 1152 may be implemented by any type of volatile or nonvolatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0171] In addition to instructions 1153 , memory 1152 may also store data, such as road maps, route information, and data such as the location, direction, and speed of the vehicle. The data stored in memory 1152 may be used by computing platform 1150 .

[0172] In the embodiment of the present disclosure, the first processor 1151 may execute instruction 1153 to complete all or part of the steps of the above-mentioned data processing method.

[0173] In another exemplary embodiment, a computer program product is also provided. The computer program product includes a computer program executable by a programmable device, and the computer program has a code portion for executing the above data processing method when executed by the programmable device.

[0174] Some embodiments of the present disclosure also provide a chip, such as Fig.12 As shown, the chip includes at least one second processor 1201 and at least one interface circuit 1202. The second processor 1201 and the interface circuit 1202 can be interconnected through lines. For example, the interface circuit 1202 can be used to receive signals from other devices (such as the memory of the electronic device). For another example, the interface circuit 1202 can be used to send signals to other devices (such as the second processor 1201). Exemplarily, the interface circuit 1202 can read the instructions stored in the memory and send the instructions to the second processor 1201. When the instructions are executed by the second processor 1201, the data processing device can execute the various steps in the above embodiments. Of course, the chip can also include other discrete devices, and some embodiments of the present disclosure are not specifically limited to this.

[0175] In some embodiments of the present disclosure, the interface circuit 1202 can obtain data, program instructions and / or information from the internal storage area of ​​the chip; it can also obtain data, program instructions and / or information from outside the chip.

[0176] Optionally, the chip may also include a memory for storing necessary computer programs and data.

[0177] Those skilled in the art may also understand that the various illustrative logical blocks and steps listed in the embodiments of the present application may be implemented by electronic hardware, computer software, or a combination of the two. Whether such functions are implemented by hardware or software depends on the specific application and the design requirements of the entire system. Those skilled in the art may use various methods to implement the functions described for each specific application, but such implementation should not be understood as exceeding the scope of protection of the embodiments of the present application.

Claims

1. A data processing method, characterized in that: include: Control the first acquisition device to send a synchronization signal to the second acquisition device at a first time point; The synchronization signal instructs the second acquisition device to acquire image data of the target object upon receiving the synchronization signal, and there is a communication delay time between the first acquisition device and the second acquisition device; Control the first acquisition device to acquire point cloud data of the target object at a second time point; the interval between the first time point and the second time point is equal to the communication delay time; The step of controlling the first acquisition device to send a synchronization signal to the second acquisition device at a first time point includes: When the position of the rotating mirror in the first acquisition device reaches the first position, it is determined that the first time point has been reached; and the first acquisition device is controlled to send the synchronization signal to the second acquisition device at the first time point.

2. The method according to claim 1, characterized in that The controlling the first acquisition device to acquire point cloud data of the target object at a second time point includes: When the position of the rotating mirror in the first acquisition device reaches a second position, determining that the second time point has been reached; Control the first acquisition device to acquire the point cloud data at the second time point.

3. The method according to claim 1, characterized in that: The method further comprises: The first time point is determined according to the second time point and the communication delay duration.

4. The method according to claim 3, characterized in that When the rotating mirror position in the first acquisition device reaches a first position, it indicates that the first time point has been reached; when the rotating mirror position in the first acquisition device reaches a second position, it indicates that the second time point has been reached; The determining the first time point according to the second time point and the communication delay duration includes: Obtaining a rotation angle of the rotating mirror according to the communication delay time and the rotation angular velocity of the rotating mirror; The first position is determined according to the second position and the rotation angle.

5. The method according to claim 4, characterized in that The method further comprises: Obtaining a turning speed of the rotating mirror according to a frequency at which the first acquisition device acquires the point cloud data and the number of mirror surfaces of the rotating mirror; The rotation angular velocity is obtained according to the steering speed.

6. The method according to claim 1, characterized in that The controlling the first acquisition device to send a synchronization signal to the second acquisition device at a first time point includes: The first acquisition device is controlled to forward the synchronization signal to the second acquisition device through a controller at the first time point.

7. The method according to claim 6, characterized in that The method further comprises: The communication delay duration is obtained according to a first data transmission duration between the first acquisition device and the controller, a second data transmission duration between the second acquisition device and the controller, and a third data transmission duration for data transmitted within the controller.

8. A data processing device, characterized in that: include: A synchronization module, configured to control the first acquisition device to send a synchronization signal to the second acquisition device at a first time point; The synchronization signal instructs the second acquisition device to acquire image data of the target object upon receiving the synchronization signal, and there is a communication delay time between the first acquisition device and the second acquisition device; A collection module, configured to control the first collection device to collect point cloud data of the target object at a second time point; the interval between the first time point and the second time point is equal to the communication delay time; The synchronization module is further configured to determine that the first time point has been reached when the rotating mirror position in the first acquisition device reaches a first position; and control the first acquisition device to send the synchronization signal to the second acquisition device at the first time point.

9. A chip, characterized in that: The chip includes a processing unit and an interface circuit, the processing unit obtains program instructions through the interface circuit, the program instructions are executed by the processing unit, and the processing unit is used to execute the steps of the method as described in any one of claims 1 to 7.

10. A data acquisition device, characterized in that: The data acquisition device is configured with the chip as claimed in claim 9.

11. A vehicle, characterized in that: The vehicle is provided with the data acquisition device as claimed in claim 10.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 7 are implemented.

Images