Method and encoder for encoding lidar data

The method and encoder enable the technical application.

CN119087391AActive Publication Date: 2024-12-06AXIS
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202410680628.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-06-05
Filing Date
2024-05-29
Publication Date
2024-12-06
Estimated Expiration
2044-05-29

AI Technical Summary

Technical Problem

Existing LiDAR data encoding methods are inefficient in handling multiple return signals per emitted laser ray, leading to large data volumes that are difficult to compress effectively.

Method used

A method and encoder that represent LiDAR data as video frames, stacking different return signals in image portions based on parameters like distance, Doppler shift, or reflectivity, and apply video coding techniques to encode these frames efficiently.

Benefits of technology

This approach allows for efficient encoding and compression of LiDAR data, reducing the data volume while maintaining the ability to recognize objects, objects, and their parameters, and enhancing the efficiency of the technical efficacy of the technical solution, and enabling effective encoding and compression of the data without compromising the ability to recognize objects from the technical application.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119087391A_ABST
    Figure CN119087391A_ABST
Patent Text Reader

Abstract

A method and an encoder for encoding lidar data are provided. A subsequent frame of lidar data to be encoded is received. Each frame of the lidar data includes a plurality of lidar return signals for each of a plurality of rays emitted by the lidar at respective elevation and azimuth angles, and each lidar return signal includes a lidar measurement. Then, each frame of the lidar data is represented as an image frame 306-i of the video sequence, where, for each of the plurality of rays of the frame of the lidar data, a lidar measurement of a different lidar return signal is represented in a different image portion 602-1, 602-2, 602-3 of the image frame 306-i. The different image portions 602-1, 602-2, 602-3 are stacked behind each other in a row direction 606 or a column direction 604 of the image frame 306-i. The video sequence is then encoded using video coding.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of encoding of laser radar data. In particular, the present invention relates to an encoding method and an encoder for encoding laser radar data. Background Art

[0002] LiDAR technology has been used successfully in the automotive industry for many years and has more recently been seen in the surveillance industry as a complement to surveillance cameras and radar. LiDAR is able to create high-resolution three-dimensional images that can be used to detect the distance, reflectivity, direction, and speed of objects in its field of view. Compared to cameras, LiDAR is less dependent on the lighting conditions in the scene, which makes it attractive for surveillance in low-light conditions.

[0003] The laser radar operates to repeatedly scan the monitored scene with laser light by emitting laser rays at different elevation and azimuth angles, and records various parameters of any return signal returned to the laser radar. Laser rays emitted at specific azimuth and elevation angles may not generate a return signal, generate one or several return signals. Each return signal in turn results in the measurement of one or more physical parameters such as the distance, reflectivity and speed (Doppler shift) of the object in the scene that reflects the laser beam to generate the return signal. Therefore, the laser radar generates a huge amount of data when repeatedly scanning the scene. For example, it may generate a data volume of about 100MB / s.

[0004] Due to the size of the lidar data, it is preferably compressed before transmission and storage. For example, it would be desirable to compress the data from approximately 100 MB / s to 5 MB / s by a factor of 20. It has been suggested in US2020 / 0394822A1 to represent the lidar data as a two-dimensional color image and to compress the lidar data using image or video compression techniques. On a similar topic, US2022 / 0292722A1 has suggested converting a point cloud into an image representation by segmenting the point cloud into blocks of continuous subsets of the surface described by the point cloud. These blocks are then projected onto corresponding projection planes to generate block images, which in turn are packed into image frames. However, there is still room for improvement, particularly with regard to how to effectively handle the fact that an emitted laser ray may result in several return signals. Summary of the invention

[0005] In view of the above, it is therefore an object of the present invention to alleviate the above problems and provide a way to efficiently encode lidar data, especially when the transmitted ray may result in more than one return signal.

[0006] This object is achieved by the invention as defined by the appended independent claims.Example embodiments are defined by the appended dependent claims.

[0007] According to a first aspect of the present invention, the above-mentioned purpose is achieved by a method for encoding lidar data. The method comprises: receiving subsequent frames of lidar data to be encoded, wherein each frame of the lidar data comprises a plurality of lidar return signals for each of a plurality of rays emitted by the lidar at corresponding elevation and azimuth angles, and wherein each lidar return signal comprises a lidar measurement value; representing each frame of the lidar data as an image frame of a video sequence, wherein for each of a plurality of rays of the frame of the lidar data, a lidar measurement value of a different lidar return signal is represented in a different image portion of the image frame, wherein the different image portions are stacked behind each other in a row direction or a column direction of the image frame, wherein each image portion has a pixel grid corresponding to an elevation angle and an azimuth angle and a pixel value formed by the lidar measurement value of the lidar return signal represented in the image portion, wherein the pixel value of the image portion is formed by the lidar measurement value by interpolating the lidar measurement values ​​of the elevation angle and the azimuth angle from the plurality of rays to the pixel grid of the image portion; and encoding the video sequence using video coding.

[0008] Using this method, different return signals are represented in different image portions that are stacked behind each other to form an image frame. In this way, a frame of lidar data can be efficiently represented and encoded as a single image frame, even when some of the transmitted rays result in more than one return signal.

[0009] As described above, lidar data is typically collected by repeatedly scanning a scene with a laser by emitting laser rays at different elevation and azimuth angles, and recording various parameters of any return signals returning to the lidar. As used herein, a frame of lidar data refers to the lidar data collected during one such scan of a scene. Accordingly, a sequence of frames of lidar data is generated as the lidar repeats the scan of the scene. Sometimes, a scan of the lidar is also referred to as a sweep.

[0010] For each laser ray emitted by the lidar, multiple lidar return signals may be recorded in the lidar data. Multiple may be zero, one, or several lidar return signals. For example, if the lidar is pointed at a clear sky where no reflecting objects exist, there may be no return signal, whereas when there are several reflecting objects in the path of the laser ray, there may be several return signals. In this article, it is assumed that at least one of the emitted laser rays results in several lidar return signals.

[0011] Different image parts being stacked behind each other in a row direction or a column direction of the image frame means that the image parts are arranged or positioned behind each other in an image plane of the image frame. The row direction refers to the direction in which pixel rows of the image frame in the image plane extend, and the column direction refers to the direction in which pixel columns of the image frame in the image plane extend.

[0012] Representing a frame of lidar data as an image frame generally means that the lidar data in the frame is converted or mapped to an image frame. For example, the elevation and azimuth of a shot ray can be converted or mapped to pixel locations of an image frame, and lidar measurements can be converted or mapped to pixel values ​​of an image frame. Representing lidar measurements in an image portion of an image generally means that the lidar measurements are converted or mapped to pixel values ​​of pixels in that portion of the image.

[0013] Coding efficiency is further affected by the selection of which return signal has its lidar measurement value represented in which image portion. For example, in order to take advantage of the temporal encoding provided by video coding techniques, it is preferred that the return signals generated by particular objects in the scene are arranged in the same image portion in the current image frame and in temporally adjacent image frames. If so, lidar measurements such as the distance, reflectivity, and speed of the object can be efficiently temporally encoded by referencing backward to the corresponding values ​​in the previous image frame. One way to achieve this is to determine, for each of a plurality of rays of a frame of lidar data, which lidar return signal has its lidar measurement value represented in which image portion, depending on the lidar measurement value of the lidar return signal.

[0014] For example, in one embodiment, the lidar measurement value of each lidar return signal includes a distance value, and wherein, for each of multiple rays of a frame of lidar data, the measurement value of the lidar return signal having a maximum distance value is represented in a first image portion, and the measurement value of the lidar return signal having a second largest distance value is represented in a second image portion.

[0015] In this case, the inventors have realized that for each ray, the return signal with the largest distance value generally corresponds to the background in the scene. By sorting the return signals into image portions of the image frame according to decreasing distance, the background in the scene is generally represented in the same image portion in each image frame. Therefore, it can be efficiently encoded using time encoding.

[0016] In other embodiments, determining which laser radar return signal to make its measurement value represent in which image portion depends on the similarity of laser radar measurement values ​​such as distance, reflectivity and Doppler shift of laser radar return signals of different rays. Specifically, the laser radar measurement value of each laser radar return signal may include a Doppler shift value, and wherein determining which laser radar return signal to make its measurement value represent in which image portion depends on the similarity of the Doppler shift values ​​of the laser radar return signals of different rays. This embodiment will cause laser radar return signals with similar Doppler shift values ​​to be represented in the same image portion in the image frame. Since laser radar return signals reflected from the same object usually have similar Doppler shift values, laser radar return signals from the same object will be represented in the same image portion in the image frame. As a result, the laser radar measurement value can be effectively time-encoded. In another example, determining which laser radar return signal to make its measurement value represent in which image portion depends on the similarity of the distance values ​​of the laser radar return signals of different rays. This embodiment will cause laser radar return signals with similar distance values ​​to be represented in the same image portion in the image frame.

[0017] At least two of the different image parts can be encoded as different slices or tiles. As known in the art of video coding, a slice or tile refers to a spatially distinct area of ​​an image frame that is encoded separately from any other area in the same image frame. Therefore, different image parts can be encoded separately from each other. This in turn allows different image parts to be decoded separately. Encoding as separate slices / tiles can also be advantageous because some slices / tiles can be omitted from encoding to save processing power, or can be discarded before transmission or storage to save bandwidth or storage space.

[0018] Different levels of compression may be applied to at least two of the image portions. This may be advantageous in various circumstances. For example, it is noted that the resolution of a lidar, as measured in the number of rays emitted per area, decreases with distance. Furthermore, it is also noted that at a certain distance, a certain resolution is required to be able to recognize an object. Therefore, in an embodiment where the return signals are sorted into different image portions in order of decreasing distance, it would make sense to compress the image portions corresponding to larger distances at a lower level of compression than the image portions corresponding to closer distances. In that way, the encoding becomes more efficient without compromising the ability to recognize objects from the lidar data.

[0019] According to the present invention, each image portion has a pixel grid corresponding to the elevation and azimuth angles and a pixel value formed by the laser radar measurement value of the laser radar return signal represented in the image portion. The pixel grid is preferably the same over time (i.e., in each image frame) to facilitate the comparison of pixel values ​​and time encoding over time. However, a problem that may arise is that due to tolerances in the laser radar mechanism, there may be changes in the elevation and azimuth angles of the emitted rays between different scans of the laser radar (i.e., between different frames). If the pixel grid is defined by the elevation and azimuth angles of multiple rays emitted by the laser radar during the scan, it will therefore not be constant over time. In order to deal with this problem, according to the present invention, the pixel values ​​of the image portion are formed from the laser radar measurement values ​​by interpolating the laser radar measurement values ​​of the elevation and azimuth angles from the multiple rays to the pixel grid of the image portion. In that way, the comparison of pixel values ​​and time encoding over time is facilitated.

[0020] The lidar measurement values ​​of each lidar return signal may include a measurement value of a first parameter, a measurement value of a second parameter, and a measurement value of a third parameter, wherein the measurement value of the first parameter is represented in a first color channel of the image frame, the measurement value of the second parameter is represented in a second color channel of the image frame, and the measurement value of the third parameter is represented in a third color channel of the image frame. By using a color image, different color channels may be used to represent the measurement values ​​of different parameters in the same image frame. The first parameter may correspond to a distance, the second parameter may correspond to a reflectivity, and the third parameter may correspond to a Doppler shift.

[0021] The second color channel and the third color channel may have a resolution lower than that of the first color channel. For example, because the distance parameter is considered to be most important for monitoring applications, it may be described at a higher resolution than the resolution of reflectivity and Doppler shift. Specifically, the image frame may be represented in a YUV 4:2:0 color format, which is a common format used in video encoding, thereby making the method suitable for use with existing video encoders.

[0022] In addition, existing video encoders typically support pixel values ​​at a specific bit depth, such as 8 bits or 10 bits. However, it is not uncommon for one or more measurement values ​​of lidar data, such as distance measurement values, to be represented by a higher number of bits. In other words, the lidar measurement value may include a measurement value of a first parameter represented using a first number of bits, and the pixel value in the image frame may be represented by a second number of bits lower than the first number of bits. Therefore, such a measurement value of the lidar data cannot be represented as a single pixel value of the image frame. To address this problem, for each image portion, a first subset of the first number of bits may be represented in a first sub-portion of the image portion, and a second subset of the first number of bits may be represented in a second sub-portion of the image portion. The second sub-portion of the image portion may be further stacked behind the first sub-portion of the image portion in one of the row direction or column direction of the image frame. In this way, even when the measurement value is represented by more bits than the bit depth of the image frame, the lidar measurement value of the lidar data frame can still be represented in a single image frame.

[0023] The first subset of bits and the second subset of bits may be odd bits and even bits, respectively. This has several advantages. Firstly, if instead the most significant bit is represented in the first sub-portion and the least significant bit is represented in the second sub-portion, a coding error in the first sub-portion will have an effect that may even exceed the information in the least significant bit. This risk is mitigated by dividing the bits into even bits and odd bits. A further advantage is that if further compression is desired, the second sub-portion with the even bits can be discarded, while the remaining first sub-portion still gives an acceptable resolution spatially and temporally also for lower values.

[0024] In an embodiment where the first subset of bits includes the most significant bit and the second subset of bits includes the least significant bit, another method for reducing the impact of coding errors is to append one or more auxiliary bits to the first subset of bits before encoding. Possibly, one or more auxiliary bits can also be added to the second subset of bits. Then, on the decoding side, these auxiliary bits will be removed again after decoding. When encoding, any coding error will mainly affect the least significant bit. Accordingly, in this case, the additional auxiliary bits will be mainly affected, rather than the first subset of bits corresponding to the most significant bit of the measured value of the first parameter. Therefore, it can be said that the auxiliary bits protect the most significant bits from coding errors. In that way, when encoding the first subset of bits, they act as auxiliary or support, thereby being auxiliary. Auxiliary bits can generally be regarded as additional bits attached behind the least significant bit of the first subset of bits. The value of the additional bits is predetermined.

[0025] The first sub-portion and the second sub-portion of at least one image portion may be encoded as different slices or tiles. Again, this allows different sub-portions to be encoded and decoded separately. If a higher compression ratio is desired, this further allows the sub-portions to be removed from the video after encoding.

[0026] Furthermore, when encoding the video sequence, a lower compression level may be applied to the first sub-portion compared to the second sub-portion of at least one image portion. This may be used to compensate for the fact that coding errors have a greater effect in the first sub-portion, either when the first sub-portion comprises even bits or when it comprises the most significant bits. For example, in an embodiment with odd bits in the first sub-portion and even bits in the second sub-portion, the size of coding errors in the first sub-portion (odd bits) is twice as large as errors in the second sub-portion (even bits). Therefore, a compression level that is a factor of two lower may be used in the first sub-portion compared to the second sub-portion.

[0027] According to a second aspect and a third aspect of the present invention, as defined by the independent claims, the above objects are achieved by an encoder and a non-transitory computer readable medium for encoding lidar data, respectively. The second and third aspects may generally have the same features and advantages as the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The above and additional objects, features and advantages of the present invention will be better understood through the following illustrative and non-limiting detailed description of embodiments of the present invention with reference to the accompanying drawings, in which like reference numerals are used for similar elements, and in which:

[0029] Figure 1 Illustration of the lidar used to monitor the scene.

[0030] Figure 2 Schematically illustrates a lidar scan of a scene and a sequence of frames of lidar data resulting from subsequent lidar scans of the scene according to an embodiment.

[0031] Figure 3 An encoder for encoding lidar data according to an embodiment is schematically illustrated.

[0032] Figure 4 is a flowchart of a method for encoding lidar data according to an embodiment.

[0033] Figure 5 A sequence of frames of lidar data and a corresponding sequence of image frames forming a video sequence according to an embodiment are schematically illustrated.

[0034] Figure 6 An image frame in which a frame of lidar data is represented is schematically illustrated according to an embodiment.

[0035] Figure 7 Schematically illustrated Figure 6 The image portion of the image frame.

[0036] Figure 8 An image frame in which a frame of lidar data is represented is schematically illustrated according to an embodiment. DETAILED DESCRIPTION

[0037] The present invention will now be described more fully hereinafter with reference to the accompanying drawings, in which embodiments of the invention are shown.The systems and apparatus disclosed herein will be described during operation.

[0038] Figure 1 The laser radar 100 for monitoring a scene 110 is shown. The scene 110 includes objects such as a house 112, bushes 114 in front of the house 112, and a person 116. The laser radar 100 scans the scene 110 by emitting a laser ray 102 toward the scene 110. The emitted laser ray 102 reflects off the objects in the scene 110 and returns to the laser radar 100 in the form of a return signal 104. As the laser ray 102 propagates away from the laser radar 100, its cross section increases. Therefore, one emitted laser ray 102 may result in more than one return signal 104. In the illustrated example, the laser ray 102 results in two return signals, one return signal 104-1 from the reflection in the bushes and another return signal 104-2 from the reflection in the house from a portion of the ray that has passed through the bushes. It is understood that the emitted laser ray 102 can in principle result in any number of return signals such as no return signal, one return signal, or several return signals. However, there is usually a maximum number of return signals per emitted ray that the laser radar 100 can detect. For example, there are commercially available laser radars that are capable of detecting up to three return signals or up to six return signals per emitted ray. When the return signals 104 return to the laser radar 100, they are detected. Each return signal 104 indicates various parameters or characteristics of the object reflecting the laser ray 102, such as distance, reflectivity and speed (Doppler shift). Therefore, based on each return signal 104, the laser radar 100 can generate measurements of these parameters, which are referred to as laser radar measurements in this article, and associate these measurements with the return signal 104. How the laser radar 100 generates such measurements is known per se in the art.

[0039] Table 1 below is an example of what type of data the lidar 100 generates for each transmitted ray, along with the number of bits used to represent each data item. As can be seen therein, a scan ID is provided for each ray, which indicates during which scan of the scene the ray was emitted. The time at which the ray was emitted, as well as the elevation and azimuth angles at which the ray was emitted, are also provided for each ray. In addition, measurements of various parameters of each return signal associated with the ray are provided. In this case, this includes measurements of distance, reflectivity, and Doppler shift (velocity), but there are many lidars that only measure distance and reflectivity.

[0040]

[0041] Table 1

[0042] When scanning a scene 110, the laser radar 100 emits laser rays 102 at a specific rate. Each ray 102 is emitted at an elevation angle and an azimuth angle. For example, by using one or more rotating mirrors in the laser radar 100, at least one of the elevation angle and the azimuth angle is changed between subsequent rays 102, so that the elevation angle and the azimuth angle of the emitted ray 102 form a scanning pattern. This principle is Figure 2 It is further shown in Figure 2 A scanning pattern 200 of azimuth and elevation of a plurality of rays emitted by a lidar during a scan of a scene is illustrated. Each point 202 corresponds to the azimuth and elevation of an emitted ray, and arrows 204 indicate the order in which the rays are emitted during the scan. In this example, the scan is performed in a row-by-row manner, but of course other scan patterns and scan directions are possible. However, it is understood that many other variations and options are possible. It is further understood that Figure 2 The number of rays in is chosen for illustration purposes. In a more realistic case, during a scan of a scene, the lidar may emit thousands of rays, such as 200×250 rays. The resolution of the lidar can be changed dynamically. For example, it can remain at a first resolution for a first number of subsequent frames, and then change to a second resolution for a second number of subsequent frames.

[0043] During the scan, the lidar 100 detects the return signal associated with each of the emitted rays and outputs a measurement value (e.g., as illustrated in Table 1) associated with the return signal. The result data from all rays emitted during the scan is collected in a frame 206-1 of the lidar data. Accordingly, the lidar data in frame 206-1 includes measurement values ​​of multiple lidar return signals from each ray emitted during the scan. Once the scan is over, the lidar 100 starts a new scan of the scene and generates lidar data collected in a subsequent frame 206-2 of the lidar data. This is then repeated again and again, and as a result, a sequence 206 of frames 206-1, 206-2, 206-3, 206-4 of lidar data is generated, where these frames correspond to subsequent scans of the scene.

[0044] Figure 3 An encoder 300 is shown configured to encode lidar data generated by lidar 100. Encoder 300 generally includes circuitry 302 configured to implement input interface functionality 310 as well as image frame generation functionality 320 and encoding functionality 330. In short, input interface functionality 310 is configured to receive subsequent frames of lidar data to be encoded, such as sequence 206 of frames generated by lidar 100. Image frame generation functionality 320 is configured to represent each frame of lidar data as an image frame of a video sequence, and encoding functionality 330 is configured to encode the video sequence using video encoding to provide an encoded video sequence.

[0045] In a hardware implementation, the encoder 300 may include, for each function 310, 320, 330, corresponding circuitry dedicated and specifically designed to implement that function. The circuitry may be in the form of one or more integrated circuits such as one or more application specific integrated circuits or one or more field programmable gate arrays. By way of example, the encoder 300 may therefore include circuitry that, when in use, receives a subsequent frame of lidar data to be encoded.

[0046] In a software implementation, as an alternative, the circuitry may be in the form of a processor such as a microprocessor or a central processing unit associated with computer code instructions stored on a (non-transitory) computer readable medium such as a non-volatile memory to cause the encoder 300 to perform any of the methods disclosed herein. Examples of non-volatile memory include read-only memory, flash memory, ferroelectric RAM, magnetic computer storage devices, and optical disks, etc. In a software implementation, the functions 310, 320, 330 may therefore each correspond to a portion of computer code instructions stored on a computer readable medium, which, when executed by the processor, causes the encoder 300 to perform the function.

[0047] It shall be understood that it is also possible to have a combination of hardware and software implementation, meaning that some of the functions 310, 320, 330 are implemented in hardware and others in software.

[0048] Now refer to Figures 4 to 8 And further refer to Figure 1 , Figure 2 and Figure 3 The operation of encoder 300 when performing method 400 for encoding lidar data is described.

[0049] Figure 4 is a flow chart of a method 400 for encoding lidar data. In step 402, the input interface function 310 receives a signal such as Figure 2 The sequence 206 of lidar frames includes frames 206-1, 206-2, 206-3, 206-4, 206-5, and subsequent frames of lidar data to be encoded. As described above, each frame 206-1, 206-2, 206-3, 206-4, 206-5 includes data for each ray emitted by the lidar 100 during the scanning of the scene 110. Specifically, it includes lidar measurements associated with multiple lidar return signals for each emitted ray. For example, the lidar measurements may include measurements of distance, reflectivity, and / or velocity (Doppler shift). In another example, the lidar measurements include measurements of distance and reflectivity. In yet another example, the lidar measurements include measurements of distance. In addition, as shown in Table 1, it may include a scan ID, emission time, elevation, and azimuth for each emitted ray.

[0050] The input interface function 310 forwards the received frames 206-1, 206-2, 206-3, 206-4, 206-5 of the lidar data to the image frame generation function 320. In step 404, the image frame generation function 304 represents each frame of the lidar data as an image frame of a video sequence. Figure 5As shown in , the lidar data frame 206-1 is represented in the image frame 306-1 of the video sequence 306, the lidar data frame 206-2 is represented in the subsequent image frame 306-2 of the video sequence 306, and so on. In the frame header of each image frame 306-1 to 306-5, the input interface function 310 may include an identifier of the corresponding lidar data frame, such as an identifier of the lidar data frame and its corresponding scan (refer to the scan ID of Table 1). In that way, the identifier is sent once per frame, rather than once per ray. In addition, the frame header may include the start time of the lidar data frame, and optionally the end time. Alternatively, the end time can be calculated based on the start time of the next frame. Since lidars typically emit rays at a known rate, once the start time and end time of the frame are known, the time of each ray can be derived. In that way, the time information is only sent once in the frame header, rather than once per ray.

[0051] Figure 6 The image frame 306-i generated by the image frame generation function 320 is illustrated in more detail. As shown, the image frame 306-i has multiple image parts, in this case illustrated by three image parts 602-1, 602-2, 602-3. The number of image parts corresponds to the maximum number of return signals that the laser radar 100 can detect per ray. Continuing with the example from Table 1 above, the number of image parts can therefore be three or six. In this case, the image parts 602-1, 602-2, 602-3 are stacked (i.e., arranged) behind each other in the column direction 604 of the image frame 306-i. However, it will also be possible to stack the image parts 602-1, 602-2, 602-3 behind each other in the row direction 606 of the image frame 306-i.

[0052] When a frame of lidar data is represented as an image frame 306-i, lidar measurements of different return signals associated with the transmitted ray are represented in different image portions 602-1, 602-2, 602-3 of the image frame 306-i. By way of example, assume that for a first transmitted ray, two lidar return signals are detected. Thus, a lidar measurement of one of the lidar return signals may be represented in the first image portion 602-1, and the other lidar return signal may be represented in the second image portion 602-2. Since there are only two lidar return signals for the first ray, no lidar measurement is represented in the third image portion 602-1 for the first ray. Furthermore, it is said that for a second transmitted ray, one lidar return signal is detected. Thus, a lidar measurement of the lidar return signal may be represented in the first image portion 602-1, while no lidar measurement of the second ray is represented in the second image portion 602-2 and the third image portion 602-3.

[0053] When the frame of the laser radar data is represented as an image frame 306-i, the image frame generation component 320 therefore determines which laser radar return signal to represent its laser radar measurement value in which image portion 602-1, 602-2, 602-3 of the image frame 306-i. In order to make this determination, the image frame generation component 320 can use different methods. A naive method is to sort the laser radar return signals of the rays into different image portions 602-1, 602-2, 602-3 in the order in which they appear in the frame of the laser radar data. That is, the measurement value of the laser radar return signal that appears for the first time is represented in the first image portion 601-1, the measurement value of the laser radar return signal that appears for the second time is represented in the second image portion 601-2, and so on. In more complex methods, the determination depends on the laser radar measurement value of the laser radar return signal. For example, the determination can depend on the distance, reflectivity and / or Doppler shift of the laser radar return signal. As will be described below, by applying these more complex methods, additional bit rate savings are achieved.

[0054] In one embodiment, the lidar return signals for each ray are sorted into different image portions 602-1, 602-2, 602-3 depending on the distance. Specifically, starting from the return signal with the largest measured distance, the return signals are sorted into different image portions 602-1, 602-2, 602-3 in order of decreasing distance. Therefore, for each ray, the measurement value of the lidar return signal with the largest distance value is represented in the first image portion 602-1. If there is more than one lidar return signal for the ray, the measurement value of the lidar return signal with the second largest distance value is represented in the second image portion 602-2, and so on. The reasoning behind this approach is that the return signal associated with the largest distance is typically a reflection from the background of the scene, while return signals with a shorter distance are typically caused by noise such as raindrops or snowflakes. With this approach, reflections from the background will therefore appear in the same image portion such as the first image portion 602-1 in each image frame. Since the background typically remains stable over time, this allows for efficient time encoding of the first image portion 602-1. In Figure 6 , where in the first image portion 602-1 the return signals originating from the house 112 and the person 116 are represented, while in the second image portion 602-2 and the third image portion 602-3 various reflections from the bushes 114 in front of the house 112 are represented. Generally, the level of noise will also be greater in the image portions corresponding to the return signals with a shorter distance than in the image portions with the return signals with a longer distance.

[0055] In another embodiment, instead, the laser radar return signals of the emitted rays are sorted into different image parts 602-1, 602-2, 602-3 depending on their Doppler shift values. Specifically, determining which laser radar return signal to have its measurement value represented in which image part depends on the similarity of the Doppler shift values ​​of the laser radar return signals of different rays. For example, the return signals are sorted into different image parts 602-1, 602-2, 602-3 so that return signals with similar Doppler shift values ​​will have their measurement values ​​represented in the same image part in the current image frame 306-i and in other image frames of the video sequence 306. In this case, if two Doppler shift values ​​deviate from each other by less than a predetermined threshold, they can be said to be similar. This can be achieved, for example, by representing return signals having Doppler shift values ​​within a first predetermined interval in the first image portion 602-1, representing return signals having Doppler shift values ​​within a second predetermined interval in the second image portion 602-2, and representing return signals having a third Doppler shift value in the third image portion 602-3. In this embodiment, all return signals from objects moving at a certain speed will generally end up in the same image portion of the image frame, thereby allowing efficient use of spatial and temporal video encoding of the object. For example, all return signals originating from a person 116 will end up in the same image portion. This embodiment is particularly advantageous when the lidar is used to monitor scenes such as traffic monitoring in which there are many moving objects.

[0056] In yet another embodiment, the laser radar return signals of the emitted rays are sorted into different image portions 602-1, 602-2, 602-3 depending on their reflectivity values. Even if the laser radar return signals are emitted from the same object in the scene, their reflectivity values ​​will vary depending on the angle of incidence. However, changes in reflectivity values ​​originating from the same object will typically vary continuously. Therefore, the laser radar return signals can be sorted into different image portions 602-1, 602-2, 602-3 depending on the continuity of the reflectivity values, so that laser radar return signals with reflectivity values ​​that exhibit continuous changes are sorted into the same image portion.

[0057] When the image frame generation function 320 has determined which lidar return signals should have their lidar measurements represented in which image portion 602-1, 602-2, 602-3, the lidar measurements are then represented as image data in those image portions 602-1, 602-2, 602-3. Figure 7 This is explained. Figure 7One image portion is illustrated, here image portion 602-1, but it should be understood that the same applies to all image portions. Image portion 602-1 has a pixel grid 702. Pixel grid 702 is the same for each image portion and is also fixed over time, so that pixel grid 702 is the same in all image frames 306-1 to 306-5 of video sequence 306. Pixel grid 702 corresponds to a regular grid of elevation and azimuth angles. Therefore, each pixel defined by pixel grid 702 corresponds to an elevation and azimuth angle value. In the example shown in FIG. Figure 2 There is typically not a one-to-one mapping between the scan pattern of emitted rays illustrated in FIG. 1 and the grid of elevation and azimuth values ​​corresponding to the pixel grid 702 and its grid. The reason for not using the elevation and azimuth of the emitted rays directly as the pixel grid is that the elevation and azimuth of the emitted rays do not necessarily form a regular grid, and additionally they will vary slightly from scan to scan and are therefore not the same in each frame of the lidar data. Also, the grid of elevation and azimuth defined by the pixel grid 702 covers the scan pattern of elevation and azimuth of the emitted rays. Preferably, the number of pixels in each of the elevation and azimuth directions of the pixel grid 702 is higher, such as doubled, compared to the scan pattern of the emitted rays for reasons of avoiding aliasing problems when performing interpolation as will be described below. By way of example, if the lidar in each scan emits rays in a scan pattern of 200×250, a 400×500 pixel grid may be used.

[0058] The pixel values ​​in image portion 602-1 are formed from the lidar measurements of the radar return signal to be represented in image portion 602-1. To form the pixel values ​​of image portion 602-1, image frame generation function 320 interpolates the lidar measurements of the elevation and azimuth angles from the transmitted rays to pixel grid 702 of image portion 602-1. Referring again to Figure 7 , points 704 schematically illustrate the elevation and azimuth angles associated with those return signals that should have their lidar measurements represented in image portion 602-1. As can be seen, the points are not perfectly aligned with pixel grid 702, and there is no one-to-one mapping between the elevation and azimuth angles of the lidar return signals and the pixels of pixel grid 702. Therefore, interpolation is used to map the lidar measurements of the lidar return signals to the pixels of pixel grid 702. For this purpose, any conventional interpolation method such as linear interpolation or spline interpolation may be used.

[0059] As previously described, the laser radar measurement value of each laser radar return signal may include measurement values ​​of various parameters including a first parameter (distance), a second parameter (reflectivity) and / or a third parameter (Doppler shift). In order to represent all the measurement values ​​in a single image frame, different color channels 706-1, 706-2, and 706-3 can be used. For example, the measurement value of the first parameter can be represented in the first color channel 706-1, the measurement value of the second parameter can be represented in the second color channel 706-2, and the measurement value of the third parameter can be represented in the third color channel 706-3. The first color channel can correspond to the brightness (Y) channel, and the second color channel and the third color channel can correspond to the chrominance channels (U, V) of the YUV color format commonly used in video encoding. Therefore, the measurement value of the first parameter of the laser radar return signal is interpolated to the pixel grid 702, and the result is arranged in the first color channel 706-1. The same applies to the second parameter and / or the third parameter, and the results are arranged in the second color channel 706-2 and the third color channel 706-3, respectively. Thus, after interpolation, each pixel in image portion 602-1 includes interpolated values ​​for different parameters in different color channels.

[0060] A specific version of the YUV color format common in video encoding is the YUV 4:2:0 color format. In some embodiments, the image frame generation function 320 therefore represents the frames 206-1, 206-2, 206-3, 206-4, 206-5 of the lidar data in the YUV 4:2:0 color format. In the YUV color format, the spatial resolution in the second color channel and the third color channel (U, V) is lower than the spatial resolution in the first color channel (Y). In more detail, the resolution in the chrominance channels U and V is four times lower (halved in each of the row and column directions) than the luminance channel Y. Accordingly, the second color channel 706-2 and the third color channel 706-3 of the image portion 602-1 can have a resolution four times lower than the resolution of the first color channel 706-1. This means that the pixel grid 702 in the second color channel 706-2 and the third color channel 706-3 has a resolution four times lower than the resolution of the pixel grid 702 in the first color channel 706-1. By way of example, if a 400x500 pixel grid is used in the first color channel 706-1, a 200x250 pixel grid is used in the other two color channels 706-2, 706-3.

[0061] It is more common in video encoding to work with a 10-bit or 8-bit YUV 4:2:0 color format (i.e., the pixel value in each channel is represented by 10 or 8 bits). As can be seen in Table 1 above, some of the measurements in the LiDAR data can be represented by more than 10 or 8 bits. We will now describe how this can be done in such a case.

[0062] In the following example, it is assumed that the pixel values ​​of the image frame 306-i are represented using a second number of bits such as 8 bits or 10 bits, and the lidar measurement value includes a measurement value represented using a first number of bits such as 16 bits or 19 bits. Then, the first number of bits used to represent the measurement value is divided into a subset of a number of bits including a first subset and a second subset. This can be considered as dividing a pixel into a number of pixels including different subsets of bits of the original pixel. As will be explained below, these pixels will be located in different sub-portions of the image portion. The pixel division is usually performed after the measurement value has been interpolated to the pixel grid of the image portion as described above. Each subset of bits includes at most a second number of bits, so that the subset can be represented as a pixel value in the image frame 306-i. For example, if the measurement value is represented by 16 bits, it can be divided into two subsets of 8 bits each, so that each subset can be represented by a pixel value having 8 bits. It is understood that sometimes two subsets are sufficient, while in other cases, more subsets are needed to make the number of bits in the subset meet the limit set by the second number of bits. As an alternative to having more subsets, the first number of bits may be reduced by setting some bits to zero before dividing them into subsets. For example, in Table 1, the distance measurement is represented by 19 bits, and it may be possible to reduce the distance measurement to, say, 16 bits by setting the first and / or last bit to zero depending on whether short or large distances are most important in the application at hand. By way of example, with 19 bits, distances within an interval of 2 mm to 1000 m may be represented, while with 16 bits, distances within an interval of 5 mm to 500 m may be represented.

[0063] In the case where the lidar data includes measurements of more than one parameter, the above-mentioned division of the bits representing the measurements into subsets is applied to each measurement. Each measurement is divided into the same number of subsets. For example, it is assumed that the lidar data uses 16 bits to represent the distance, 10 bits to represent the reflectivity, and 8 bits to represent the Doppler shift. Thus, the distance value can be divided into two subsets each having 8 bits, the reflectivity can be divided into two subsets each having 5 bits, and the Doppler shift can be divided into two subsets each having 4 bits. The first of these subsets will be arranged in different color channels of the first sub-portion of the image portion, and the second of these subsets will be arranged in different color channels of the second sub-portion of the image portion. As understood, in this example, there are many alternative ways to divide the bits into subsets. For example, alternatively, the Doppler shift can be divided into a first subset having 8 bits and a second empty subset having no bits. In this case, the second sub-portion of the image portion will be empty, such as including a null value or a zero value in the color channel.

[0064] For a measurement value to be represented in a specific image portion of the image frame 306-i, different subsets of bits are represented in different sub-portions of the image portion. In particular, a first subset of a first number of bits is represented in a first sub-portion of the image portion, and a second subset of the first number of bits is to be represented in a second sub-portion of the image portion. Figure 8 This is further illustrated in Figure 8 It is shown that each image portion 602-1, 602-2, 602-3 of the image frame 306-i has multiple sub-portions for accommodating different subsets of the first number of bits. In this case, image portion 602-1 has two sub-portions 602-11 and 602-12, image portion 602-2 has two sub-portions 602-21 and 602-22, and image portion 602-3 has two sub-portions 602-31 and 602-32. It is understood that the number of sub-portions is the same as the number of subsets of the first number of bits used to represent the measurement value.

[0065] The different sub-portions 602-11, 602-12 of the image portion 602-1 have corresponding pixel grids, meaning that for each pixel in the first sub-portion 602-11, there is a pixel in the second sub-portion 602-12 corresponding to the same elevation and azimuth. In other words, the pixel grid of the second sub-portion is a copy of the pixel grid of the first sub-portion. In particular, pixels having the same relative position (such as the top left position) in the first sub-portion 602-11 and in the second sub-portion 602-12 correspond to the same elevation and azimuth. The pixel values ​​in these pixels are derived from the same measurement of the lidar data, but include different subsets of the bits of the measurement previously described.

[0066] The sub-portions are stacked, i.e. arranged behind each other in one of the row direction 606 and the column direction 604 of the image frame 306-i. In this case, the second sub-portions 602-12, 602-22, 602-32 are stacked behind the corresponding first sub-portions 602-11, 602-21, 602-31 in the row direction. It would also be possible to stack them behind each other in the column direction, so that the order of the sub-portions in the column direction would be 602-11, 602-12, 602-21, 602-22, 602-31, 602-32. An embodiment can also be envisaged in which both the image portion and its sub-portions are stacked in the row direction 606. However, from a coding point of view, image frames with more equal sizes in the row direction and in the column direction are generally more efficient. Therefore, if the image portions 602-1, 602-2, 602-3 are stacked behind each other in one of the row direction and the column direction, then Figure 8 As illustrated in , the sub-portions of each image portion are preferably stacked behind each other in the other of the row direction and the column direction.

[0067] There are different methods for how to divide the first number of bits used to represent the measured value into subsets including a first subset and a second subset. In one method, the first subset includes odd bits (bit 1, bit 3, bit 5, etc.) and the second subset includes even bits (bit 0, bit 2, bit 4, etc.). If more than two subsets are needed, this method can be generalized to arrange every i-th bit into the i-th subset.

[0068] In another method, the first subset of bits includes the most significant bit, and the second subset of bits includes the least significant bit. By way of example, if the measured value is represented by 10 bits, five most significant bits may be included in the first subset, and 5 least significant bits may be included in the second subset. If more than two subsets of bits are needed, the method is generalized to arrange the bits of the first number into subsets so that the subsets include bits in decreasing order of importance. In this case, since the coding error in the subset of the most significant bit will have a greater impact than the coding error in the subset of the least significant bit, it may be advantageous to attach one or more auxiliary bits to the first subset with the most significant bit before encoding. For example, two, three or four auxiliary bits may be added. These auxiliary bits will act as buffers to reduce the impact of coding errors. Auxiliary bits may have predefined values ​​to facilitate identification and removal of them on the decoder side. For example, it is assumed that the measured value is represented by twelve bits, which are divided into two subsets each having six bits. For each subset, two auxiliary bits having a value of zero are attached so that the resulting subset has eight bits. After encoding and subsequent decoding, the last two bits of each subset are removed and the two subsets are recombined to reconstruct the measurement value.

[0069] Once the image frame generation function 320 has represented each frame 206-1 to 206-5 of the lidar data as an image frame 306-1 to 306-5 of the video sequence 306, the encoding function 330 then encodes the video sequence 306 using video encoding. For this purpose, it can use standard video encoding tools such as h.264, h.265 or AV1. Such video encoding tools generally provide for dividing image frames into slices and / or tiles that can be encoded and decoded separately from each other. In this case, at least two of the different image portions 602-1, 602-2, 602-3 can be encoded as different slices or tiles. For example, each image portion 602-1, 602-2, 602-3 can be encoded as a different slice. In another example, two of the image portions (such as image portions 602-1, 602-2) are encoded as one slice, and the remaining image portion 602-3 is encoded as another slice. The image parts 602-1, 602-2, 602-3 in turn include several sub-parts 602-11, 602-12, 602-21, 602-22, 602-31, 602-32. Figure 8 In the same embodiment as in the embodiment of the present invention, different sub-portions can be encoded as different slices or tiles. For example, the first sub-portion 602-11 and the second sub-portion 602-12 of the image portion 602-1 can be encoded as different tiles. Depending on the desired granularity, the same can apply to all image portions 602-1, 602-2, 602-3 or only to some of them.

[0070] In addition, when encoding the video sequence 306, the video encoding function 320 may apply different levels of compression to at least two of the different image portions 602-1, 602-2, 602-3. The level of compression may correspond to a quantization parameter (QP) used in the compression. In one example, a lower level of compression is applied to image portions in which return signals that are farther away have their measurements represented than to image portions in which return signals that are shorter in distance have their measurements represented. Figure 6 and Figure 8 In an embodiment, the lowest compression level will then be applied to the image portion 602-1 corresponding to the return signal with the highest distance value, an intermediate compression level will be applied to the image portion 602-2 corresponding to the return signal with the second highest distance value, and the highest compression level will be applied to the image portion 602-3 corresponding to the third highest distance value.

[0071] The compression level may also be set to be different between different sub-portions of the image portions 602-1, 602-2, 602-3 to account for the fact that coding errors may have different effects in different sub-portions. For example, in an embodiment where odd bits (bit 1, bit 3, bit 5, etc.) are arranged in a first sub-portion and even bits (bit 0, bit 2, bit 4, etc.) are arranged in a second sub-portion, errors in the first sub-portion are doubled in size compared to errors in the second sub-portion. Therefore, a lower compression level such as halving the compression error may be used in the first sub-portion compared to the second sub-portion. In an embodiment where the most significant bit is arranged in the first sub-portion and the least significant bit is arranged in the second sub-portion, a lower compression level is preferably used in the first sub-portion compared to the second sub-portion. In the case where auxiliary bits are appended to the first sub-portion having the most significant bit, the compression level in the first sub-portion may be set so that the maximum possible compression error does not exceed the maximum number that can be represented by the auxiliary bits.

[0072] The encoding function 330 can encode the video sequence 306 according to a group of pictures (GOP) structure. In the case where the resolution of the lidar data changes dynamically, the GOP can be synchronized with the change in the lidar resolution. For example, if the lidar provides a first resolution during a first number of subsequent lidar frames and provides a second resolution during a second number of subsequent lidar frames, the image frames corresponding to the first number of lidar frames can be encoded as a first GOP, and the image frames corresponding to the second number of lidar frames can be encoded as a second GOP.

Claims

1. A method for encoding lidar data, comprising: receiving subsequent frames of lidar data to be encoded, wherein each frame of the lidar data comprises a plurality of lidar return signals for each of a plurality of rays emitted by the lidar at corresponding elevation and azimuth angles, and wherein each lidar return signal comprises a lidar measurement value; Represent each frame of lidar data as an image frame of a video sequence, wherein, for each of the plurality of rays of the frame of the lidar data, lidar measurements representing different lidar return signals in different image portions of the image frame, wherein the different image parts are stacked behind each other in the row direction or the column direction of the image frame, wherein each image portion has a pixel grid corresponding to elevation and azimuth angles and pixel values ​​formed by the lidar measurements of the lidar return signals represented in the image portion, wherein the pixel values ​​of the image portion are formed by the lidar measurements by interpolating the lidar measurements of the elevation and azimuth angles from the plurality of rays to the pixel grid of the image portion; and The video sequence is encoded using video coding.

2. The method according to claim 1, wherein: For each of the plurality of rays of the frame of the lidar data, it is determined which lidar return signal to have its lidar measurement value represented in which image portion depending on the lidar measurement value of the lidar return signal.

3. The method according to claim 1, wherein: The lidar measurement value of each lidar return signal includes a distance value, and wherein, for each of the multiple rays of the frame of the lidar data, the measurement value of the lidar return signal with the largest distance value is represented in the first image portion, and the measurement value of the lidar return signal with the second largest distance value is represented in the second image portion.

4. The method according to claim 2, wherein: The lidar measurement value of each lidar return signal includes a Doppler shift value, and wherein the determination of which lidar return signal to have its measurement value represent in which image portion depends on the similarity of the Doppler shift values ​​of the lidar return signals of different rays.

5. The method according to claim 1, wherein: At least two of the different image parts are encoded as different slices or tiles.

6. The method according to claim 1, wherein: When encoding the video sequence, different levels of compression are applied to at least two of the image portions.

7. The method according to claim 1, wherein: The lidar measurement value of each lidar return signal includes a measurement value of a first parameter, a measurement value of a second parameter, and a measurement value of a third parameter, wherein the measurement value of the first parameter is represented in a first color channel of the image frame, the measurement value of the second parameter is represented in a second color channel of the image frame, and the measurement value of the third parameter is represented in a third color channel of the image frame.

8. The method according to claim 1, in, The lidar measurements include measurements of a first parameter represented using a first number of bits, and wherein pixel values ​​in the image frame are represented by a second number of bits less than the first number of bits, wherein, for each image portion, a first subset of said first number of bits is represented in a first sub-portion of said image portion, and a second subset of said first number of bits is represented in a second sub-portion of said image portion, and Wherein, the second sub-portion of the image portion is stacked behind the first sub-portion of the image portion in one of the row direction or the column direction of the image frame.

9. The method according to claim 8, wherein: The first subset of bits and the second subset of bits are odd bits and even bits, respectively.

10. The method according to claim 8, wherein: The first subset of bits comprises most significant bits and the second subset of bits comprises least significant bits, and wherein one or more auxiliary bits are appended to the first subset of bits prior to encoding.

11. The method according to claim 8, wherein: The first sub-portion and the second sub-portion of at least one image portion are encoded as different slices or tiles.

12. The method according to claim 8, wherein: When encoding the video sequence, a lower level of compression is applied to the first sub-portion than to the second sub-portion of at least one image portion.

13. An encoder for encoding lidar data, comprising a circuit configured to: An input interface function configured to receive subsequent frames of lidar data to be encoded, wherein: Each frame of the lidar data comprises a plurality of lidar return signals for each of a plurality of rays emitted by the lidar at respective elevation and azimuth angles, and wherein each lidar return signal comprises a lidar measurement value; an image frame generation function configured to represent each frame of the lidar data as an image frame of a video sequence, wherein, for each of the plurality of rays of the frame of the lidar data, lidar measurements representing different lidar return signals in different image portions of the image frame, wherein the different image parts are stacked behind each other in the row direction or the column direction of the image frame, wherein each image portion has a pixel grid corresponding to elevation and azimuth angles and pixel values ​​formed by the lidar measurements of the lidar return signals represented in the image portion, wherein the pixel values ​​of the image portion are formed by the lidar measurements by interpolating the lidar measurements of the elevation and azimuth angles from the plurality of rays to the pixel grid of the image portion; and The encoding function is configured to encode the video sequence using video coding.

14. A non-transitory computer readable medium comprising computer code instructions which, when executed by a device having processing capabilities, cause the device to perform the method according to claim 1.

Citation Information

Patent Citations

  • Video encoding / decoding methods, corresponding computer programs and video encoding / decoding devices

    CN104519363A

  • Laser radar data processing method and system

    CN114265035A

  • Laser point cloud data processing method and device, storage medium and equipment

    CN115586541A

  • Method and apparatus for point cloud compression

    US20200394822A1

  • Occupancy map block-to-patch information compression

    US20220292722A1