Data processing system, data processing method, and program

Abstract

A data processing system comprises: a first complementing unit that, on the basis of first depth coordinate data in a first frame and first reflectance data indicating the reflectance of each coordinate of the first depth coordinate data, complements a first missing portion in the first depth coordinate data; and a motion compensation unit that compensates for motion in the complemented first depth coordinate data by using a motion vector based on the complemented first depth coordinate data and second depth coordinate data in a second frame, the motion vector indicating the difference between the first depth coordinate data and the second depth coordinate data.

Description

Data processing system, data processing method, and program

[0001] The present disclosure relates to a data processing system, a data processing method, and a program.

[0002] Point cloud data generated by a three-dimensional sensor such as LIDAR (Light Detection and Ranging) is used in various applications. Since point cloud data includes the three-dimensional coordinates of each point and has a larger data volume than a two-dimensional image or the like, compression may be required depending on the application. As a related technique, for example, Patent Document 1 is known.

[0003] International Publication No. 2020 / 183839

[0004] In Patent Document 1, it is described that a data compression device performs a compression process on point cloud data output from a LIDAR arranged in an automobile or the like. However, since it has not been considered to compress the point cloud data output from the LIDAR in time series using related techniques such as Patent Document 1, it is difficult to effectively compress the point cloud data.

[0005] In view of such problems, one of the objectives of the present disclosure is to provide a data processing system, a data processing method, and a program capable of effectively compressing point cloud data.

[0006] A data processing system according to an aspect of the present disclosure includes a first complementing unit that complements a first missing portion in the first depth coordinate data based on the first depth coordinate data in a first frame and first reflectance data indicating the reflectance of each coordinate of the first depth coordinate data, and a motion compensation unit that performs motion compensation on the complemented first depth coordinate data using a motion vector indicating a difference between the first depth coordinate data and the second depth coordinate data based on the complemented first depth coordinate data and the second depth coordinate data in a second frame.

[0007] A data processing system according to one aspect of the present disclosure includes: a first interpolation unit that interpolates a first missing portion in the first depth coordinate data based on first depth coordinate data in a first frame and first reflectance data indicating the reflectance of each coordinate in the first depth coordinate data; a second interpolation unit that interpolates a second missing portion in the second depth coordinate data based on second depth coordinate data in a second frame and second reflectance data indicating the reflectance of each coordinate in the second depth coordinate data; and a motion vector estimation unit that estimates a motion vector based on the interpolated first depth coordinate data and the interpolated second depth coordinate data.

[0008] A data processing method according to one aspect of the present disclosure includes: filling in a first missing portion in the first depth coordinate data based on first depth coordinate data in a first frame and first reflectance data indicating the reflectance of each coordinate in the first depth coordinate data; and performing motion compensation on the filled-in first depth coordinate data using a motion vector indicating the difference between the first depth coordinate data and the second depth coordinate data based on the filled-in first depth coordinate data and second depth coordinate data in a second frame.

[0009] A data processing method according to one aspect of the present disclosure includes: filling in a first missing portion in the first depth coordinate data based on first depth coordinate data in a first frame and first reflectance data indicating the reflectance of each coordinate in the first depth coordinate data; filling in a second missing portion in the second depth coordinate data based on second depth coordinate data in a second frame and second reflectance data indicating the reflectance of each coordinate in the second depth coordinate data; and estimating a motion vector based on the filled-in first depth coordinate data and the filled-in second depth coordinate data.

[0010] A program according to one aspect of the present disclosure is a program for causing a computer to perform the following processes: filling in a first missing portion in the first depth coordinate data based on first depth coordinate data in a first frame and first reflectance data indicating the reflectance of each coordinate in the first depth coordinate data; and performing motion compensation on the filled first depth coordinate data using a motion vector indicating the difference between the first depth coordinate data and the second depth coordinate data based on the filled first depth coordinate data and second depth coordinate data in a second frame.

[0011] A program according to one aspect of the present disclosure is a program for causing a computer to perform the following processes: filling in a first missing portion in the first depth coordinate data based on first depth coordinate data in a first frame and first reflectance data indicating the reflectance of each coordinate in the first depth coordinate data; filling in a second missing portion in the second depth coordinate data based on second depth coordinate data in a second frame and second reflectance data indicating the reflectance of each coordinate in the second depth coordinate data; and estimating a motion vector based on the filled-in first depth coordinate data and the filled-in second depth coordinate data.

[0012] According to this disclosure, point cloud data can be effectively compressed.

[0013] This is a diagram showing examples of compression and decompression processing according to several embodiments. This is a diagram showing examples of motion vector estimation processing in the study example. This is a diagram showing examples of motion vector estimation processing according to several embodiments. This is a configuration diagram showing examples of the configuration of a data processing system according to several embodiments. This is a configuration diagram showing examples of the configuration of a data processing system according to several embodiments. This is a configuration diagram showing examples of the configuration of a data processing device according to several embodiments. This is a flowchart showing examples of the configuration of a data processing device according to several embodiments. This is a flowchart showing examples of data processing methods according to several embodiments. This is a flowchart showing examples of data processing methods according to several embodiments. This is a configuration diagram showing examples of the configuration of a vehicle-infrastructure cooperative system according to several embodiments. This is a configuration diagram showing examples of the configuration of a terminal according to several embodiments. This is a configuration diagram showing examples of the configuration of a MEC according to several embodiments. This is a diagram showing examples of point cloud data according to several embodiments. This is a configuration diagram showing examples of the configuration of an encoder according to several embodiments. This is a diagram showing examples of depth image interpolation according to several embodiments. This is a configuration diagram showing examples of the configuration of a decoder according to several embodiments. This is a flowchart showing examples of terminal operation according to several embodiments. This is a flowchart showing examples of MEC operation according to several embodiments. This is a diagram showing examples of point cloud data according to several embodiments. This is a configuration diagram showing examples of encoder configuration according to several embodiments. This is a configuration diagram showing examples of encoder configuration according to several embodiments. This is a configuration diagram showing examples of decoder configuration according to several embodiments. This is a configuration diagram showing examples of encoder configuration according to several embodiments. This is a configuration diagram showing examples of decoder configuration according to several embodiments. This is a configuration diagram showing examples of computer hardware configurations according to several embodiments.

[0014] The embodiments will be described below with reference to the drawings. In each drawing, the same elements are denoted by the same reference numerals, and redundant explanations will be omitted where necessary.

[0015] (Overview of the Embodiment) In recent years, deep learning-based compression methods, or codecs, have been actively researched. Deep learning-based compression methods have shown improved compression efficiency for various types of data, such as images, videos, and point clouds, compared to compression methods that did not apply machine learning.

[0016] In compression methods prior to machine learning, engineers had to develop the compression method down to the smallest detail and tune it manually. In these pre-machine learning compression methods, the decoder side was standardized, and differentiation was performed on the encoder side.

[0017] In deep learning-based compression methods, most of the functionality of the compression method before machine learning is applied is replaced by a neural network, and the entire process from encoder to decoder is trained to maximize compression performance.

[0018] The inventors investigated applying CNN (Convolutional Neural Network)-based compression techniques, also used in video compression, to point cloud data (depth images), and specifically, applying CNN-based time-series compression techniques to time-series point cloud data. By utilizing information from previously compressed frames using motion vectors within the CNN-based time-series point cloud data compression, it becomes possible to reduce the amount of data. A motion vector represents changes in the position of a sensed object between frames, expressed as a vector. It can also be said that a motion vector shows the difference between two frames.

[0019] Figure 1 shows examples of compression and decompression processing according to several embodiments. The compression and decompression processing 900 in Figure 1 is an example of applying motion vector-based time-series compression in video to LIDAR point cloud data. The compression and decompression processing 900 includes the entire process from compression to decompression of point cloud data, and includes processing performed by an encoder and a decoder.

[0020] For example, when compressing a time-series frame of point cloud data including depth images, the encoder performs motion vector estimation, compression, and decompression (901). The encoder estimates the motion vector based on the current frame to be compressed and the previous frame that was previously restored. The encoder compresses and encodes the estimated motion vector, and then decompresses and restores the compressed motion vector.

[0021] Next, the encoder performs motion compensation (902) using the recovered motion vector. The encoder performs motion compensation based on the previously recovered previous frame and the recovered motion vector, and generates a motion-compensated frame. Motion compensation is the process of predicting the next frame by reflecting the estimated motion vector in the frame when performing compression using inter-frame prediction techniques.

[0022] Next, the encoder performs residual compression and decompression (903) using the motion-compensated frame. The encoder compresses and encodes the residual (difference) between the current frame and the motion-compensated frame, and then decompresses the compressed residual to restore the current frame based on the restored residual and the motion-compensated frame. The restored current frame is used for encoding the next frame.

[0023] Furthermore, the decoder reconstructs the point cloud data using the motion vectors and residuals encoded by the encoder. That is, the decoder performs an encoded motion vector decompression process (901). The decoder decompresses and reconstructs the motion vectors compressed by the encoder.

[0024] Next, the decoder performs motion compensation (902) using the recovered motion vector. The decoder performs motion compensation based on the previously recovered previous frame and the recovered motion vector, and generates a motion-compensated frame.

[0025] Next, the decoder performs a decompression (903) of the encoded residual. The decoder decompresses the residual compressed by the encoder and reconstructs the current frame based on the recovered residual and the motion-compensated frame. The reconstructed current frame is used to decode the next frame.

[0026] Thus, motion vector-based time series compression involves motion vector estimation and motion compensation. Figure 2 shows an example of motion vector estimation processing in a study considered by the inventor. As shown in Figure 2, motion vector estimation processing (901) is performed based on the previously restored previous frame and the current frame to be compressed. Motion vector estimation is performed by matching the depth and feature map regions of the previous frame and the current frame. However, in the case of depth images of point cloud data, since LIDAR indicates depth based on reflected light from objects, data may be lost depending on the reception state of the reflected light. In the case of video, the image color is continuous and motion estimation is likely to succeed, but in the case of depth images, discontinuities that occur around missing areas make matching difficult, and there is a possibility that the motion vector cannot be estimated properly and the estimation may fail. The area around missing areas refers to, for example, the area adjacent to the missing area. Similarly, when performing motion compensation, there is a possibility that motion compensation cannot be performed properly.

[0027] Therefore, in this embodiment, missing portions in the depth image of the point cloud data are interpolated. Figure 3 shows an example of motion vector estimation processing according to several embodiments. As shown in Figure 3, before performing the motion vector estimation processing (901), missing portions of the vibration image of the previous frame that was previously restored and the depth image of the current frame to be compressed are interpolated (904). For example, the interpolation of depth coordinates is performed using the reflectance included in the point cloud data. By performing motion vector estimation based on the interpolated vibration image of the previous frame and the interpolated depth image of the current frame, the motion vector can be appropriately estimated, and the success rate of motion estimation can be increased. Similarly, when performing motion compensation, it is possible to perform motion compensation appropriately by interpolating the depth coordinates.

[0028] (Embodiment 1) Next, Embodiment 1 will be described. In this embodiment, several basic examples of embodiments will be described.

[0029] Figure 4 shows an example configuration of a data processing system 10 according to several embodiments. The data processing system 10 is a system for compressing or decompressing point cloud data.

[0030] The data processing system 10 may also be a system that compresses depth coordinate data included in point cloud data generated by a LIDAR sensor using a predetermined compression method. In other words, the data processing system 10 may also be an encoding system that encodes depth coordinate data included in point cloud data using a predetermined encoding method. Note that "compression" and "encoding" are interchangeable.

[0031] Furthermore, the data processing system 10 may also be a system that decompresses compressed data using a predetermined decompression method. In other words, the data processing system 10 may also be a decoding system that decodes compressed data using a predetermined decoding method. Note that "decompression" and "decoding" are interchangeable.

[0032] In the example shown in Figure 4, the data processing system 10 includes a first interpolation unit 11 and a motion compensation unit 12. In other words, Figure 4 shows an example configuration when motion compensation is performed in a system that performs compression or decompression.

[0033] The first interpolation unit 11 interpolates the first missing portion in the first depth coordinate data based on the first depth coordinate data in the first frame and the first reflectance data indicating the reflectance of each coordinate in the first depth coordinate data. For example, when compressing or decompressing a time-series frame obtained from a LiDAR, the frame preceding the frame to be compressed or decompressed is the first frame, and the frame to be compressed or decompressed is the second frame. Either frame may be designated as the first frame or the second frame.

[0034] The first interpolation unit 11 may interpolate the first missing location in the first depth coordinate data based on the reflectance around the first missing location in the first reflectance data. In this case, the first missing location in the first depth coordinate data may be interpolated using the depth around the first missing location in the first depth coordinate data. Alternatively, the first interpolation unit may interpolate the first missing location in the first depth coordinate data using a CNN.

[0035] Furthermore, the first interpolation unit 11 may interpolate the first missing portion in the first depth coordinate data based on first mask data indicating the first missing portion in the first depth coordinate data. The mask data may be obtained from LIDAR or generated based on depth coordinate data and reflectance data.

[0036] The first interpolation unit 11 may interpolate not only the first depth coordinate data, but also the first missing portion in the first reflectance data. For example, the first interpolation unit 11 may interpolate the first missing portion in the first reflectance data based on the first reflectance data.

[0037] The motion compensation unit 12 performs motion compensation on the interpolated first depth coordinate data using a motion vector that shows the difference between the first depth coordinate data and the second depth coordinate data, based on the interpolated first depth coordinate data and the second depth coordinate data in the second frame. If the data processing system 10 performs compression, the data processing system 10 may estimate the motion vector.

[0038] Figure 5 shows an example configuration of a data processing system 20 according to several embodiments. Figure 5 shows an example configuration when estimating motion vectors in a compression system.

[0039] In the example shown in Figure 5, the data processing system 20 includes a first interpolation unit 11, a second interpolation unit 13, and a motion vector estimation unit 14, similar to those in Figure 4. The data processing system 20 may also include the motion compensation unit 12 shown in Figure 4.

[0040] The second interpolation unit 13 interpolates the second missing portion in the second depth coordinate data. The interpolation method by the second interpolation unit 13 is the same as that of the first interpolation unit 11. That is, the second interpolation unit 13 interpolates the second missing portion in the second depth coordinate data based on the second depth coordinate data in the second frame and the second reflectance data indicating the reflectance of each coordinate in the second depth coordinate data.

[0041] The motion vector estimation unit 14 estimates a motion vector based on the complemented first depth coordinate data and the complemented second depth coordinate data. For example, the motion vector estimation unit 14 may estimate the motion vector using a CNN.

[0042] Note that the data processing systems 10 and 20 may be constituted by one device or may be constituted by a plurality of devices.

[0043] FIG. 6 shows a configuration example of a data processing apparatus 30 according to some embodiments. FIG. 6 is a configuration example of a device corresponding to the data processing system 10 of FIG. 4, and may be, for example, an encoder or a decoder. In the example of FIG. 6, the data processing apparatus 30 includes the first complementation unit 11 and the motion compensation unit 12 shown in FIG. 4.

[0044] For example, part or all of the first complementation unit 11 and the motion compensation unit 12 may be arranged in a LIDAR sensor or a terminal device connected to the LIDAR sensor, or may be arranged in an edge MEC (Mobile Edge Computing) or a cloud server. The first complementation unit 11 and the motion compensation unit 12 may be distributed and arranged in a plurality of devices.

[0045] FIG. 7 shows a configuration example of a data processing apparatus 40 according to some embodiments. FIG. 7 is a configuration example of a device corresponding to the data processing system 20 of FIG. 5, and may be, for example, an encoder. In the example of FIG. 7, the data processing apparatus 40 includes the first complementation unit 11, the second complementation unit 13, and the motion vector estimation unit 14 shown in FIG. 5.

[0046] For example, part or all of the first complementation unit 11, the second complementation unit 13, and the motion vector estimation unit 14 may be arranged in a LIDAR sensor or a terminal device connected to the LIDAR sensor. The first complementation unit 11, the second complementation unit 13, and the motion vector estimation unit 14 may be distributed and arranged in a plurality of devices.

[0047] FIG. 8 shows an example of a data processing method according to some embodiments. For example, the data processing method of FIG. 8 may be executed by the data processing system 10 of FIG. 4 or the data processing apparatus 30 of FIG. 6.

[0048] In the example of FIG. 8, the first complementing unit 11 complements the first missing portion in the first depth coordinate data based on the first depth coordinate data in the first frame and the first reflectance data indicating the reflectance of each coordinate of the first depth coordinate data (S11).

[0049] Next, the motion compensation unit 12 performs motion compensation on the complemented first depth coordinate data using a motion vector indicating the difference between the first depth coordinate data and the second depth coordinate data based on the complemented first depth coordinate data and the second depth coordinate data in the second frame (S12).

[0050] FIG. 9 shows an example of a data processing method according to some embodiments. For example, the data processing method of FIG. 9 may be executed by the data processing system 20 of FIG. 5 or the data processing device 40 of FIG. 7.

[0051] In the example of FIG. 9, the first complementing unit 11 complements the first missing portion in the first depth coordinate data based on the first depth coordinate data in the first frame and the first reflectance data indicating the reflectance of each coordinate of the first depth coordinate data (S11).

[0052] Further, the second complementing unit 13 complements the second missing portion in the second depth coordinate data based on the second depth coordinate data in the second frame and the second reflectance data indicating the reflectance of each coordinate of the second depth coordinate data (S13).

[0053] Next, the motion vector estimation unit 14 estimates a motion vector based on the complemented first depth coordinate data and the complemented second depth coordinate data. Further, motion compensation may be performed on the first depth coordinate data based on the complemented first depth coordinate data and the estimated motion vector.

[0054] As described above, in the present embodiment, when compressing or expanding time-series frames using a motion vector, the missing portions of the depth coordinate data are complemented based on the depth coordinate data and the reflectance data in the frame. Thereby, since motion compensation and motion vector estimation can be appropriately performed, the point cloud data can be effectively compressed or expanded.

[0055] The following embodiments will describe specific examples of Embodiment 1.

[0056] (Embodiment 2) Next, Embodiment 2 will be described. In this embodiment, an example of supplementing missing depth images based on depth images and reflectance images will be described.

[0057] Figure 10 shows an example configuration of a vehicle-infrastructure cooperative system 1 according to several embodiments. For example, the vehicle-infrastructure cooperative system 1 is a system that supports the safety of vehicle operation by combining and analyzing information collected from roadside sensors and vehicle-side sensors. The vehicle-infrastructure cooperative system 1 may be a remote monitoring system such as an ITS (Intelligence Transport System) that monitors roads and vehicles, or a remote control system that controls vehicles according to the monitoring results. The vehicle may be an automobile, motorcycle, heavy machinery such as a forklift, a train, a robot, a drone, etc.

[0058] In the example shown in Figure 10, the vehicle-infrastructure cooperation system 1 includes multiple terminals 100, MEC 200, base stations 300, and a central server 400.

[0059] Terminal 100, MEC 200, and base station 300 are located on the road side and vehicle side (also called the road-vehicle side), while the central server 400 is located on the central side. For example, the central server 400 is located in a data center or similar location away from the road-vehicle side. For example, the road-vehicle side is the edge side of the system, and the central side is also the cloud side.

[0060] Terminal 100 and base station 300 are connected via network NW1 for communication. Network NW1 may be a wireless network such as 4G, LTE (Long Term Evolution), local 5G / 5G, other generations of mobile communication, or wireless LAN. For example, network NW1 may be a DSRC (Dedicated Short Range Communication) network for ITS systems or a V2X (Vehicle to Everything) network that connects vehicles to everything. V2X may be LTE-V2X (Long Term Evolution-V2X), NR-V2X (New Radio V2X), C-V2X (Cellular V2X), etc. Note that network NW1 is not limited to a wireless network but may also be a wired network.

[0061] The base station 300 and the MEC 200 are connected in a way that enables communication using any communication method. It can also be said that the terminal 100 and the MEC 200 are connected in a way that enables communication via the base station 300. The base station 300 and the MEC 200 may be a single device. For example, the base station 300 may have the functions of the MEC 200.

[0062] The base station 300 and the central server 400 are connected via network NW2, enabling communication. Network NW2 includes, for example, core networks such as 5GC (5th Generation Core network) and EPC (Evolved Packet Core), and the Internet. Note that network NW2 is not limited to a wired network, but may also be a wireless network. It can also be said that the terminal 100 and the central server 400 are connected via base station 300 and network NW2, enabling communication. Similarly, the MEC 200 and the central server 400 are connected via base station 300 and network NW2, enabling communication. Note that the MEC 200 and the central server 400 may also be connected via network NW2.

[0063] Terminal 100 is a terminal device connected to the network NW1. Terminal 100 is also a transmitting device that transmits sensor data measured by roadside sensors. For example, terminal 100 may be a roadside unit (RSU) installed on the roadside or an on-board unit (OBU) mounted on a vehicle. For example, roadside sensors are 3D sensors such as LiDAR sensors that generate point cloud data. Roadside sensors are not limited to LiDAR and may include cameras, etc.

[0064] For example, terminal 100 acquires point cloud data measured by a LIDAR sensor 101 installed on the vehicle-side, and transmits the acquired point cloud data to MEC 200 via base station 300. The LIDAR sensor 101 may be located outside or inside terminal 100. For example, the LIDAR sensor 101 may have the functions of terminal 100. Terminal 100 may also transmit point cloud data to center server 400 via base station 300. Terminal 100 may also transmit video data captured by a camera to MEC 200 or center server 400. Terminal 100 may also receive control information, etc., from MEC 200 or center server 400.

[0065] Terminal 100 compresses the point cloud data from the LIDAR sensor 101 using a predetermined compression method and transmits the compressed data to the MEC 200. That is, terminal 100 includes an encoder that encodes the point cloud data using a predetermined encoding method. For example, terminal 100 compresses the point cloud data using a learning model such as a CNN. Note that compression may be performed using other neural networks such as transformers, not just CNNs.

[0066] The base station 300 is a base station device for network NW1 and also a relay device that relays communication between terminal 100 and MEC200 or central server 400. For example, the base station 300 may be a local 5G base station, a 5G gNB (next generation node B), an LTE eNB (evolved node B), a wireless LAN access point, or any other relay device.

[0067] MEC200 is an edge processing unit located at the edge of the system. MEC200 may be one or more physical computers or virtual computers built on a virtualization platform. MEC200 may process sensor data received from terminal 100 and transmit the processed data to the central server 400, or it may control terminal 100 as needed. For example, MEC200 may receive point cloud data from terminal 100 via base station 300 and transmit the received point cloud data to the central server 400. MEC200 may also receive video data from terminal 100 and transmit the received video data to the central server 400.

[0068] The MEC200 receives compressed data from the terminal 100, decompresses the received compressed data using a predetermined decompression method, and restores the point cloud data. In other words, the MEC200 includes a decoder that decodes the received compressed data using a predetermined decoding method. For example, the MEC200 decompresses the compressed data using a learning model such as a CNN. Note that it is not limited to CNNs; other neural networks such as transformers may also be used for decompression.

[0069] The center server 400 is a server installed on the central side of the system. The center server 400 may be one or more physical servers, or it may be a virtualized server built on a virtualization platform. The center server 400 may also be a cloud server built on the cloud. The center server 400 monitors the situation around the vehicle by analyzing and recognizing sensor data, including point cloud data and video data from the vehicle side, and controls the vehicle as needed.

[0070] In the example shown in Figure 10, the central server 400 includes a sensor fusion function 401, a recognition function 402, a prediction function 403, etc. The sensor fusion function 401 integrates point cloud data from multiple LIDAR sensors and video data from multiple cameras to generate a 3D digital twin (3D integrated data) of the road and vehicle side.

[0071] The recognition function 402 recognizes objects on the roadside by inputting the generated 3D digital twin into the recognition AI (Artificial Intelligence) engine. For example, it recognizes vehicles, pedestrians, fallen objects, animals, etc., around the road. The prediction function 403 predicts the movement of the vehicle and the situation of pedestrians, fallen objects, animals, etc., around the moving vehicle by inputting the information of the recognized objects into the prediction AI engine.

[0072] The center server 400 may feed back the predicted results to the vehicle's terminal 100. The center server 400 may also transmit predicted information about the area around the vehicle, or transmit control information to control the vehicle's movement. For example, it can provide information about obstacles in blind spots that the vehicle cannot recognize. Even when the vehicle cannot recognize its surroundings at night or in bad weather, it can safely control automatic driving. Dangerous events that cannot be recognized by a single sensor can be reliably grasped in the 3D space of the 3D digital twin and provided to the vehicle, thereby preventing serious accidents.

[0073] Figure 11 shows an example configuration of terminal 100 according to several embodiments. Figure 12 shows an example configuration of MEC 200 according to several embodiments. Note that Figures 11 and 12 are examples of the configuration of each device, and other configurations are also acceptable as long as the operation according to some embodiments is possible. For example, some functions of terminal 100 or MEC 200 may be placed in the central server 400 or other devices.

[0074] In the example shown in Figure 11, the terminal 100 includes an acquisition unit 110, an encoder 120, and a transmission unit 130.

[0075] The acquisition unit 110 acquires point cloud data measured by the LIDAR sensor 101. For example, the LIDAR sensor 101 generates point cloud data for one frame per scan, and the acquisition unit 110 acquires the point cloud data for each frame.

[0076] Figure 13 shows examples of point cloud data according to several embodiments. In the examples in Figure 13, the point cloud data includes a depth image and a reflectance image. The depth image is coordinate information of each point in three-dimensional space obtained by reflected light from an object, and is depth coordinate data indicating the depth of each point. For example, the depth image may represent the depth of each point with an arbitrary color. The reflectance image is reflectance data indicating the reflectance of light at each point corresponding to the depth image. For example, similar to the depth image, the reflectance image may represent the reflectance of each point with an arbitrary color.

[0077] The encoder 120 encodes the depth image of the acquired point cloud data using a predetermined encoding scheme. The encoder 120 is a compression unit that compresses the depth image of the point cloud data using a predetermined compression method. The encoder 120 performs motion vector estimation and motion compensation for each frame of the acquired time series and compresses the depth image of each frame. The encoder 120 may also compress the reflectance image of each frame.

[0078] For example, the encoder 120 compresses each frame using coding with a trained CNN or entropy coding. Entropy coding may be Huffman coding, ANS (Asymmetric Numeral Systems), or other coding methods. CNN training is performed end-to-end, including the encoder 120 and the decoder 220 described later. For example, the CNNs in the encoder 120 and decoder 220 are trained to reduce data degradation and increase compression ratio.

[0079] The transmitting unit 130 transmits the encoded data encoded by the encoder 120, i.e., the compressed data, to the MEC 200. The transmitting unit 130 is a communication interface with the base station 300, and transmits the compressed data to the MEC 200 via the base station 300.

[0080] In the example shown in Figure 12, the MEC200 includes a receiving unit 210, a decoder 220, and a MEC processing unit 230.

[0081] The receiving unit 210 receives encoded data, i.e., compressed data, which is the point cloud data encoded from the terminal 100. The receiving unit 210 is a communication interface with the base station 300 and receives the compressed data from the terminal 100 via the base station 300.

[0082] The decoder 220 decodes the received compressed data using a predetermined decoding method. The decoder 220 is a decompression unit that decompresses the compressed data using a predetermined decompression method and restores the depth image of the point cloud data. The decoder 220 performs motion compensation for each received frame and restores the depth image of each frame. The decoder 220 may also restore the reflectance image of each frame. Similar to the encoder 120, the decoder 220 decompresses each frame by decoding using a trained CNN or by entropy decoding.

[0083] The MEC processing unit 230 performs predetermined MEC processing on the point cloud data, including the restored depth image and reflectance image. For example, if the point cloud data is processed by the central server 400, the restored point cloud data is sent to the central server 400. The MEC processing unit 230 may perform analysis, recognition processing, etc., on the restored point cloud data and send the processing results to the terminal 100.

[0084] Figure 14 shows an example configuration of an encoder 120 according to several embodiments. In the example in Figure 14, the encoder 120 includes a current frame interpolation unit 1201, a previous frame interpolation unit 1202, a motion vector estimation unit 1203, a motion vector compression quantization unit 1204, an entropy coding unit 1205, a motion vector extension unit 1206, a motion compensation unit 1207, a residual / reflectance compression quantization unit 1208, an entropy coding unit 1209, a depth image extension unit 1210, and a frame storage unit 1211.

[0085] The current frame interpolation unit 1201 interpolates missing portions in the depth image of the current frame to be compressed. For example, the current frame interpolation unit 1201 corresponds to the second interpolation unit 13 in Figure 5. The current frame interpolation unit 1201 interpolates missing portions in the depth image of the current frame based on the depth image and reflectance image of the current frame acquired from the LIDAR sensor 101. The current frame interpolation unit 1201 may also interpolate missing portions in the depth image and reflectance image of the current frame based on the depth image and reflectance image of the current frame. For example, the current frame interpolation unit 1201 interpolates the depth image of the current frame using a CNN that has been trained to interpolate missing portions in the depth image of the current frame according to the depth image and reflectance image of the current frame.

[0086] Figure 15 shows examples of depth image interpolation according to several embodiments. For example, the depth of a missing area is predicted using the reflectance of the missing area or the area around the missing area. For example, if there are more low reflectances than high reflectances in the missing area, it is highly likely that the missing area is a point of the same object (e.g., an object at a nearby location) as the low reflectance area. For this reason, the missing area in the depth image may be interpolated using the depth of the area in the depth image corresponding to the low reflectance area around the missing area. For example, the missing area in the depth image may be interpolated by the average depth of the low reflectance area around the missing area. Alternatively, the missing area in the reflectance image may be interpolated by the average reflectance of the low reflectance area around the missing area. The current frame interpolation unit 1201 may interpolate the missing area using a CNN that has learned the interpolation method shown in Figure 15. The current frame interpolation unit 1201 is not limited to a CNN; it may also interpolate the missing area based on the reflectance around the missing area, as shown in Figure 15.

[0087] The previous frame interpolation unit 1202 interpolates missing portions in the reconstructed depth image of the previous frame prior to the current frame. The reconstructed depth image is the depth image extracted and reconstructed by the depth image extraction unit 1210. For example, the previous frame interpolation unit 1202 corresponds to the first interpolation unit 11 in Figures 4 and 5. The previous frame interpolation unit 1202 interpolates missing portions in the reconstructed depth image of the previous frame based on the reconstructed depth image and reflectance image of the previous frame stored in the frame storage unit 1211. The previous frame interpolation unit 1202 may also interpolate missing portions in the reconstructed depth image and reflectance image of the previous frame based on the reconstructed depth image and reflectance image of the previous frame. The interpolation method of the previous frame interpolation unit 1202 is the same as that of the current frame interpolation unit 1201. For example, the previous frame interpolation unit 1202 interpolates the reconstructed depth image of the previous frame using a CNN that has been trained to interpolate missing portions in the reconstructed depth image of the previous frame according to the reconstructed depth image and reflectance image of the previous frame. The previous frame interpolation unit 1202, like the current frame interpolation unit 1201, may interpolate missing parts based on the reflectance around the missing parts, not limited to CNN.

[0088] The motion vector estimation unit 1203 estimates motion vectors based on the interpolated depth image of the current frame, interpolated by the current frame interpolation unit 1201, and the interpolated reconstructed depth image of the previous frame, interpolated by the previous frame interpolation unit 1202. The motion vectors represent the movement from the reconstructed depth image of the previous frame to the depth image of the current frame. For example, matching of each region may be performed between the reconstructed depth image of the previous frame and the depth image of the current frame, and motion vectors may be obtained according to the movement of the matched regions. For example, the motion vector estimation unit 1203 estimates motion vectors using a CNN that has been trained to estimate motion vectors according to the depth image of the current frame and the reconstructed depth image of the previous frame.

[0089] When the current frame interpolation unit 1201 and the previous frame interpolation unit 1202 interpolate depth images and reflectance images, the interpolated depth images and reflectance images may be input to the motion vector estimation unit 1203. That is, the motion vector estimation unit 1203 may estimate motion vectors based on the interpolated depth images and reflectance images of the current frame and the restored depth images and reflectance images of the previous frame.

[0090] The motion vector compression quantization unit 1204 compresses and quantizes the motion vector estimated by the motion vector estimation unit 1203. The motion vector compression quantization unit 1204 compresses (encodes) the motion vector using a predetermined compression method and quantizes the compressed motion vector to a predetermined amount of information. For example, the motion vector compression quantization unit 1204 compresses the motion vector using a CNN that has been trained to encode the motion vector.

[0091] The entropy coding unit 1205 entropy codes the motion vector that has been compressed and quantized by the motion vector compression quantization unit 1204. The entropy coding unit 1205 generates an entropy-coded motion vector code. The transmission unit 130 transmits the generated motion vector code to the MEC 200.

[0092] The motion vector decompression unit 1206 decompresses the motion vector that has been compressed and quantized by the motion vector compression quantization unit 1204. The motion vector decompression unit 1206 decompresses (decodes) the compressed and quantized motion vector using a predetermined decompression method corresponding to the compression method of the motion vector compression quantization unit 1204. For example, the motion vector decompression unit 1206 restores the motion vector using a CNN that has been trained to decode the compressed and quantized motion vector.

[0093] The motion compensation unit 1207 performs motion compensation based on the restored depth image of the previous frame after interpolation by the previous frame interpolation unit 1202 and the restored motion vector extended by the motion vector extension unit 1206. The motion compensation unit 1207 applies the motion vector to the restored depth image of the previous frame to generate a depth image after motion compensation.

[0094] The residual / reflectance compression quantization unit 1208 compresses and quantizes the residuals based on the motion-compensated depth image obtained by the motion compensation unit 1207 and the interpolated depth image of the current frame obtained by the current frame interpolation unit 1201, and also compresses and quantizes the reflectance image of the current frame. The residual / reflectance compression quantization unit 1208 calculates the residual (difference) between the motion-compensated depth image and the interpolated depth image of the current frame, compresses (encodes) the calculated residual using a predetermined compression method, and quantizes the compressed residual to a predetermined amount of information. In addition, the residual / reflectance compression quantization unit 1208 compresses (encodes) the reflectance image of the current frame using a predetermined compression method, and quantizes the compressed reflectance image to a predetermined amount of information. For example, the reflectance image is compressed and quantized using a lossless method. For example, the residual and reflectance compression quantization unit 1208 compresses the residual and reflectance images using a CNN that has been trained to encode the residual and reflectance images.

[0095] The entropy coding unit 1209 entropy codes the residual and reflectance images compressed and quantized by the residual and reflectance compression quantization unit 1208. The entropy coding unit 1209 generates the entropy-coded residual and reflectance codes. The transmission unit 130 transmits the residual and reflectance codes to the MEC 200.

[0096] The depth image decompression unit 1210 decompresses the residuals compressed and quantized by the residual / reflectance compression quantization unit 1208 to generate a restored depth image of the current frame. The depth image decompression unit 1210 decompresses (decodes) the compressed and quantized residuals using a predetermined decompression method corresponding to the compression method of the residual / reflectance compression quantization unit 1208. For example, the depth image decompression unit 1210 restores the residuals using a CNN that has been trained to decode the compressed and quantized residuals. The depth image decompression unit 1210 applies the decompressed residuals to the motion-compensated depth image, which has been motion-compensated by the motion compensation unit 1207, to restore the depth image of the current frame. For example, the depth image is compressed and decompressed using a lossy method.

[0097] The frame storage unit 1211 stores the reconstructed depth image of the current frame generated by the depth image decompression unit 1210. The frame storage unit 1211 also stores the reflectance image of the current frame. The reconstructed depth image and reflectance image stored in the frame storage unit 1211 are used by the previous frame interpolation unit 1202 as the reconstructed depth image and reflectance image of the previous frame in the compression process of the next frame.

[0098] Figure 16 shows an example configuration of a decoder 220 according to several embodiments. In the example in Figure 16, the decoder 220 includes an entropy decoding unit 2201, a motion vector extension unit 2202, a previous frame interpolation unit 2203, a motion compensation unit 2204, an entropy decoding unit 2205, a depth image extension unit 2206, a reflectance extension unit 2207, and a frame storage unit 2208.

[0099] The entropy decoding unit 2201 entropically decodes the motion vector code received by the receiving unit 210 from the terminal 100. The entropy decoding by the entropy decoding unit 2201 corresponds to the entropy coding by the entropy coding unit 1205 of the encoder 120. The entropy decoding unit 2201 entropically decodes the motion vector code and restores the motion vector after compression and quantization.

[0100] The motion vector decompression unit 2202, like the motion vector decompression unit 1206 of the encoder 120, decompresses the compressed and quantized motion vector restored by the entropy decoding unit 2201. The motion vector decompression unit 2202 decompresses (decodes) the compressed and quantized motion vector using a predetermined decompression method corresponding to the compression method of the motion vector compression quantization unit 1204 of the encoder 120. For example, the motion vector decompression unit 2202 restores the motion vector using a CNN that has been trained to decode the compressed and quantized motion vector.

[0101] The previous frame interpolation unit 2203 interpolates missing portions in the restored depth image of the previous frame, similar to the previous frame interpolation unit 1202 of the encoder 120. For example, the previous frame interpolation unit 2203 corresponds to the first interpolation unit 11 in Figure 4. The previous frame interpolation unit 2203 interpolates missing portions in the restored depth image of the previous frame based on the restored depth image and restored reflectance image of the previous frame stored in the frame storage unit 2208. The previous frame interpolation unit 2203 may also interpolate missing portions in the restored depth image and restored reflectance image of the previous frame based on the restored depth image and restored reflectance image of the previous frame. For example, the previous frame interpolation unit 2203 interpolates the restored depth image of the previous frame using a CNN that has been trained to interpolate missing portions in the restored depth image of the previous frame according to the restored depth image and restored reflectance image of the previous frame.

[0102] The motion compensation unit 2204, similar to the motion compensation unit 1207 of the encoder 120, performs motion compensation based on the restored depth image of the previous frame after interpolation by the previous frame interpolation unit 2203 and the restored motion vector extended by the motion vector extension unit 2202. The motion compensation unit 2204 applies the motion vector to the restored depth image of the previous frame to generate a depth image after motion compensation.

[0103] The entropy decoding unit 2205 entropically decodes the residual and reflectance codes received by the receiving unit 210 from the terminal 100. The entropy decoding by the entropy decoding unit 2205 corresponds to the entropy coding by the entropy coding unit 1209 of the encoder 120. The entropy decoding unit 2205 entropically decodes the residual and reflectance codes and restores the residual and reflectance images after compression and quantization.

[0104] The depth image decompression unit 2206, similar to the depth image decompression unit 1210 of the encoder 120, decompresses the residuals after compression and quantization restored by the entropy decoding unit 2205 to generate a restored depth image of the current frame. The depth image decompression unit 2206 decompresses (decodes) the residuals after compression and quantization using a predetermined decompression method corresponding to the compression method of the residual / reflectance compression quantization unit 1208 of the encoder 120. For example, the depth image decompression unit 2206 restores the residuals using a CNN that has been trained to decode the compressed and quantized residuals. The depth image decompression unit 2206 applies the decompressed residuals to the motion-compensated depth image, which has been motion-compensated by the motion compensation unit 2204, to restore the depth image of the current frame. For example, the depth image is compressed and decompressed using a lossy method.

[0105] The reflectance decompression unit 2207 decompresses the compressed and quantized reflectance image restored by the entropy decoding unit 2205 to generate a restored reflectance image of the current frame. The reflectance decompression unit 2207 decompresses (decodes) the compressed and quantized reflectance image using a predetermined decompression method corresponding to the compression method of the residual / reflectance compression quantization unit 1208 of the encoder 120. For example, the reflectance decompression unit 2207 restores the reflectance image using a CNN that has been trained to decode the compressed and quantized reflectance image. For example, the reflectance image is compressed and decompressed in a lossless manner.

[0106] The frame storage unit 2208 stores the reconstructed depth image of the current frame generated by the depth image extension unit 2206. The frame storage unit 2208 also stores the reconstructed reflectance image of the current frame generated by the reflectance extension unit 2207. The reconstructed depth image and reconstructed reflectance image stored in the frame storage unit 2208 are used by the previous frame interpolation unit 2203 as the reconstructed depth image and reconstructed reflectance image of the previous frame in the extension process of the next frame.

[0107] Figure 17 shows examples of operation of terminal 100 according to several embodiments. In the example in Figure 17, the acquisition unit 110 acquires point cloud data of the current frame measured by the LIDAR sensor 101 (S101). The point cloud data of the current frame includes a depth image and a reflectance image. The following S102 to S111 are performed by the respective parts of the encoder 120.

[0108] Next, the current frame interpolation unit 1201 interpolates the missing parts in the acquired depth image of the current frame (S102). For example, the current frame interpolation unit 1201 uses a CNN to interpolate the missing parts in the depth image of the current frame based on the acquired depth image and reflectance image of the current frame.

[0109] Furthermore, the previous frame interpolation unit 1202 interpolates any missing parts in the reconstructed depth image of the previous frame (S103). For example, the previous frame interpolation unit 1202 uses a CNN to interpolate any missing parts in the reconstructed depth image of the previous frame based on the reconstructed depth image and reflectance image of the previous frame stored in the frame storage unit 1211. Note that S102 and S103 may be executed in parallel, or one may be executed first.

[0110] Following S102 and S103, the motion vector estimation unit 1203 estimates the motion vector based on the interpolated depth image of the current frame interpolated in S102 and the interpolated reconstructed depth image of the previous frame interpolated in S103 (S104). For example, the motion vector estimation unit 1203 uses a CNN to estimate the motion vector based on the interpolated depth image of the current frame and the interpolated reconstructed depth image of the previous frame.

[0111] Next, the motion vector compression quantization unit 1204 compresses and quantizes the estimated motion vector (S105). For example, the motion vector compression quantization unit 1204 uses a CNN to compress the motion vector using a predetermined compression method and then quantizes the compressed motion vector.

[0112] Next, the entropy coding unit 1205 entropy codes the compressed and quantized motion vector (S106). For example, the entropy coding unit 1205 entropy codes the compressed and quantized motion vector to generate a motion vector code.

[0113] Furthermore, following S105, the motion vector expansion unit 1206 expands the compressed and quantized motion vector (S107). For example, the motion vector expansion unit 1206 uses a CNN to expand the compressed and quantized motion vector in a predetermined expansion method and restore the motion vector.

[0114] Next, the motion compensation unit 1207 performs motion compensation based on the restored depth image of the previous frame that was interpolated in S103 and the restored motion vector that was expanded in S107 (S108). For example, the motion compensation unit 1207 applies the motion vector to the restored depth image of the previous frame to generate a depth image after motion compensation.

[0115] Next, the residual / reflectance compression quantization unit 1208 compresses and quantizes the residuals based on the motion-compensated depth image obtained in S108 and the interpolated depth image of the current frame obtained in S102, and also compresses and quantizes the reflectance image of the current frame obtained in S101 (S109). For example, the residual / reflectance compression quantization unit 1208 calculates the residual (difference) between the motion-compensated depth image and the interpolated depth image of the current frame. The residual / reflectance compression quantization unit 1208 compresses the calculated residual and the reflectance image of the current frame using a predetermined compression method, quantizes the compressed residual and the reflectance image of the current frame, and generates the compressed and quantized residual and reflectance images.

[0116] Next, the entropy coding unit 1209 entropy codes the compressed and quantized residual and reflectance images (S110). For example, the entropy coding unit 1209 entropy codes the compressed and quantized residual and reflectance images to generate residual and reflectance codes.

[0117] Furthermore, following S109, the depth image decompression unit 1210 decompresses the residuals compressed and quantized in S109 and generates a reconstructed depth image of the current frame using the motion-compensated depth image obtained in S108 (S111). For example, the depth image decompression unit 1210 uses a CNN to decompress the residuals after compression and quantization using a predetermined decompression method, and applies the decompressed residuals to the motion-compensated depth image to reconstruct the depth image of the current frame. The frame storage unit 1211 stores the generated reconstructed depth image of the current frame. The frame storage unit 1211 also stores the reflectance image of the current frame. Subsequently, in the compression process of the next frame, interpolation processing of the previous frame (S103) is performed using the stored reconstructed depth image and reflectance image.

[0118] Furthermore, following S106 and S110, the transmitting unit 130 transmits the encoded motion vector code and residual / reflectance code to the MEC 200 (S112). The transmitting unit 130 transmits the motion vector code and residual / reflectance code together as one so that they can be recognized as codes of the same frame. For example, the motion vector code and residual / reflectance code may be transmitted in a single packet, or they may be multiplexed and transmitted using any multiplexing scheme.

[0119] Figure 18 shows examples of operation of the MEC200 according to several embodiments. In the example in Figure 18, the receiving unit 210 receives motion vector codes and residual / reflectance codes from the terminal 100 (S201). For example, the receiving unit 210 acquires motion vector codes and residual / reflectance codes from one packet. The following steps S202 to S208 are performed by the decoder 220.

[0120] Next, the entropy decoding unit 2201 entropically decodes the received motion vector code (S202). For example, the entropy decoding unit 2201 entropically decodes the motion vector code and reconstructs the compressed and quantized motion vector.

[0121] Next, the motion vector expansion unit 2202 expands the restored compressed and quantized motion vector (S203). For example, the motion vector expansion unit 2202 uses a CNN to expand the compressed and quantized motion vector in a predetermined expansion method and restore the motion vector.

[0122] Furthermore, the previous frame interpolation unit 2203 interpolates any missing parts in the reconstructed depth image of the previous frame (S204). For example, the previous frame interpolation unit 2203 uses a CNN to interpolate any missing parts in the reconstructed depth image of the previous frame based on the reconstructed depth image and reconstructed reflectance image of the previous frame stored in the frame storage unit 2208. Note that S203 and S204 may be executed in parallel, or one may be executed first.

[0123] Following S203 and S204, the motion compensation unit 2204 performs motion compensation based on the restored depth image of the previous frame after interpolation, which was interpolated in S204, and the restored motion vector, which was expanded in S203 (S205). For example, the motion compensation unit 2204 applies the motion vector to the restored depth image of the previous frame to generate a depth image after motion compensation.

[0124] Furthermore, following S201, the entropy decoding unit 2205 entropically decodes the received residual and reflectance codes (S206). For example, the entropy decoding unit 2205 entropically decodes the residual and reflectance codes and reconstructs the compressed and quantized residual and reflectance images. Note that S205 and S206 may be executed in parallel, or one may be executed first.

[0125] Following S205 and S206, the depth image decompression unit 2206 decompresses the compressed and quantized residuals restored in S206 and generates a restored depth image of the current frame using the motion-compensated depth image that was motion-compensated in S205 (S207). For example, the depth image decompression unit 2206 uses a CNN to decompress the compressed and quantized residuals in a predetermined decompression method, applies the decompressed residuals to the motion-compensated depth image, and restores the depth image of the current frame. The frame storage unit 2208 stores the generated restored depth image of the current frame.

[0126] Furthermore, following S206, the reflectance decompression unit 2207 decompresses the restored compressed and quantized reflectance image to generate a restored reflectance image of the current frame (S208). For example, the reflectance decompression unit 2207 uses a CNN to decompress the compressed and quantized reflectance image using a predetermined decompression method to restore the reflectance image of the current frame. The frame storage unit 2208 stores the generated restored reflectance image of the current frame. Subsequently, in the decompression process of the next frame, interpolation processing of the previous frame (S204) is performed using the stored restored depth image and restored reflectance image.

[0127] Furthermore, following S207 and S208, the MEC processing unit 230 performs predetermined MEC processing on the point cloud data, including the restored depth image and reflectance image (S209). For example, the MEC processing unit 230 transmits the restored point cloud data to the central server 400.

[0128] As described above, in this embodiment, in an encoder that compresses point cloud data using motion vectors and a decoder that decompresses it, missing depth images are interpolated based on depth images and reflectance images. By predicting the depth of missing areas using reflectance, missing depth images can be appropriately interpolated. By interpolating missing depth images, motion vector estimation can be performed appropriately, thus improving the probability of successful motion vector estimation. Since residuals are calculated based on interpolated frames, the residuals become smoother (continuous) and easier to compress, reducing the amount of residual code and improving overall compression efficiency.

[0129] (Embodiment 3) Next, Embodiment 3 will be described. In this embodiment, an example of supplementing missing depth coordinate data based on a mask will be described. The configuration of the vehicle-infrastructure cooperation system 1, terminal 100, and MEC 200 is the same as in Embodiment 2.

[0130] Figure 19 shows examples of point cloud data according to several embodiments. In the example in Figure 19, the point cloud data includes a depth image and a reflectance image, as well as a mask. The mask is data that indicates missing locations in the depth image. That is, the mask indicates whether or not each point in the depth image is missing. For example, the mask may represent the presence or absence of missing points in black and white. The mask may be acquired from the LIDAR sensor 101 or generated from the depth image and reflectance image.

[0131] Figures 20 and 21 show examples of the encoder 120 configuration according to several embodiments. Figure 20 shows an example of the encoder 120 configuration when acquiring a mask from a LIDAR sensor 101.

[0132] In the example shown in Figure 20, the encoder 120, as in Figure 14, includes a current frame interpolation unit 1201, a previous frame interpolation unit 1202, a motion vector estimation unit 1203, a motion vector compression quantization unit 1204, an entropy coding unit 1205, a motion vector decompression unit 1206, a motion compensation unit 1207, an entropy coding unit 1209, a depth image decompression unit 1210, and a frame storage unit 1211, and also includes a residual / reflectance / mask compression quantization unit 1208a.

[0133] This section will mainly describe a configuration that differs from that shown in Figure 14. In the example shown in Figure 20, the current frame interpolation unit 1201 interpolates missing portions of the depth image of the current frame based on the depth image, reflectance image, and mask acquired from the LIDAR sensor 101. For example, the current frame interpolation unit 1201 may identify missing portions of the depth image based on the mask and interpolate the identified missing portions of the depth image.

[0134] The previous frame interpolation unit 1202 interpolates the missing parts of the previous frame's restored depth image based on the restored depth image, reflectance image, and mask of the previous frame stored in the frame storage unit 1211. For example, the previous frame interpolation unit 1202 may identify the missing parts of the restored depth image based on the mask and interpolate the identified missing parts of the restored depth image.

[0135] The residual / reflectance / mask compression quantization unit 1208a compresses and quantizes the residuals based on the motion-compensated depth image obtained by the motion compensation unit 1207 and the interpolated depth image of the current frame obtained by the current frame interpolation unit 1201, and also compresses and quantizes the reflectance image and mask of the current frame. The residual / reflectance / mask compression quantization unit 1208a may also compress and quantize the residuals in the region where mask = 0 (no loss). For example, the reflectance image and mask are compressed and quantized in a reversible manner.

[0136] The entropy coding unit 1209 entropy codes the residual, reflectance, and mask images and masks that have been compressed and quantized by the residual, reflectance, and mask compression quantization unit 1208a. The entropy coding unit 1209 generates the entropy-coded residual, reflectance, and mask codes. The transmission unit 130 transmits the residual, reflectance, and mask codes to the MEC 200. The frame storage unit 1211 stores the restored depth image of the current frame, the reflectance image of the current frame, and the mask of the current frame.

[0137] Figure 21 shows an example configuration of the encoder 120 when generating a mask. In the example in Figure 21, the encoder 120 includes a mask generation unit 1212 in addition to the configuration in Figure 20.

[0138] The mask generation unit 1212 generates a mask based on the depth image and reflectance image of the frame acquired from the LIDAR sensor 101. For example, the mask generation unit 1212 generates a mask by treating locations where the depth in the depth image is 0 and the reflectance in the reflectance image is 0 as missing locations. The current frame interpolation unit 1201 interpolates the missing locations in the depth image of the current frame based on the depth image and reflectance image of the current frame acquired from the LIDAR sensor 101 and the mask generated by the mask generation unit 1212. The rest is the same as in Figure 20.

[0139] Figure 22 shows an example configuration of a decoder 220 according to several embodiments. In the example of Figure 22, the decoder 220, similar to Figure 16, includes an entropy decoding unit 2201, a motion vector extension unit 2202, a previous frame interpolation unit 2203, a motion compensation unit 2204, an entropy decoding unit 2205, a depth image extension unit 2206, and a frame storage unit 2208, and also includes a reflectance / mask extension unit 2207a.

[0140] This section will mainly describe a configuration that differs from that shown in Figure 16. In the example shown in Figure 22, the previous frame interpolation unit 2203 interpolates the missing portions of the restored depth image of the previous frame based on the restored depth image, restored reflectance image, and restored mask of the previous frame stored in the frame storage unit 2208. For example, the previous frame interpolation unit 2203 may identify the missing portions of the restored depth image based on the restored mask and interpolate the identified missing portions of the restored depth image.

[0141] The entropy decoding unit 2205 entropy-decodes the residual, reflectance, and mask code received by the receiving unit 210 from the terminal 100.

[0142] The reflectance / mask decompression unit 2207a decompresses the compressed and quantized reflectance image and mask restored by the entropy decoding unit 2205 to generate the reflectance image and mask of the current frame.

[0143] The frame storage unit 2208 stores the depth image restored by the depth image extension unit 2206, the reflectance image of the current frame restored by the reflectance / mask extension unit 2207a, and the mask.

[0144] (Modification of Embodiment 3) As a modification of Embodiment 3, the depth image may be interpolated based on the depth image and a mask, without using a reflectance image.

[0145] Figure 23 shows an example configuration of the encoder 120 according to several embodiments. Figure 23 is an example configuration of the encoder 120 when a reflectance image is not used, compared to the example in Figure 20.

[0146] In the example shown in Figure 23, the encoder 120, as in Figures 14 and 20, includes a current frame interpolation unit 1201, a previous frame interpolation unit 1202, a motion vector estimation unit 1203, a motion vector compression quantization unit 1204, an entropy coding unit 1205, a motion vector decompression unit 1206, a motion compensation unit 1207, an entropy coding unit 1209, a depth image decompression unit 1210, and a frame storage unit 1211, and also includes a residual / mask compression quantization unit 1208b.

[0147] This section will mainly describe a configuration that differs from those in Figures 14 and 20. In the example in Figure 23, the current frame interpolation unit 1201 interpolates missing parts of the depth image of the current frame based on the depth image and mask of the current frame acquired from the LIDAR sensor 101.

[0148] The previous frame interpolation unit 1202 interpolates the missing parts of the previous frame's restored depth image based on the restored depth image and mask of the previous frame stored in the frame storage unit 1211.

[0149] The residual / mask compression quantization unit 1208b compresses and quantizes the residuals based on the motion-compensated depth image obtained by the motion compensation unit 1207 and the interpolated depth image of the current frame obtained by the current frame interpolation unit 1201, and also compresses and quantizes the mask of the current frame.

[0150] The entropy coding unit 1209 entropy codes the residuals and mask compressed and quantized by the residual / mask compression quantization unit 1208b. The entropy coding unit 1209 generates the entropy-coded residual / mask codes. The transmission unit 130 transmits the residual / mask codes to the MEC 200. The frame storage unit 1211 stores the restored depth image of the current frame and the mask of the current frame.

[0151] Figure 24 shows an example configuration of the decoder 220 according to several embodiments. Figure 24 is an example configuration of the decoder 220 when a reflectance image is not used, compared to the example in Figure 22.

[0152] In the example shown in Figure 24, the decoder 220, as in Figures 16 and 22, comprises an entropy decoding unit 2201, a motion vector extension unit 2202, a previous frame interpolation unit 2203, a motion compensation unit 2204, an entropy decoding unit 2205, a depth image extension unit 2206, a frame storage unit 2208, and also includes a mask extension unit 2207b.

[0153] This section will mainly describe a configuration that differs from those in Figures 16 and 22. In the example in Figure 24, the previous frame interpolation unit 2203 interpolates the missing parts of the previous frame's restored depth image based on the restored depth image and restored mask of the previous frame stored in the frame storage unit 2208.

[0154] The entropy decoding unit 2205 entropy-decodes the residual and mask codes received by the receiving unit 210 from the terminal 100.

[0155] The mask extension unit 2207b extends the compressed and quantized mask restored by the entropy decoding unit 2205 to generate the mask for the current frame.

[0156] The frame storage unit 2208 stores the depth image restored by the depth image extension unit 2206 and the mask of the current frame restored by the mask extension unit 2207b.

[0157] As described above, in an encoder that compresses point cloud data using motion vectors and a decoder that decompresses it, missing depth images may be further compensated for based on a mask. By identifying missing areas and predicting depth using a mask, missing depth images can be appropriately compensated for.

[0158] This disclosure is not limited to the embodiments described above, and may be modified as appropriate without departing from its spirit.

[0159] Each configuration in the above-described embodiment may consist of hardware, software, or both, and may consist of one piece of hardware or software, or multiple pieces of hardware or software. Functions (processing) of terminals, MECs, etc. may be realized by a computer 50 having a processor 51 such as a CPU (Central Processing Unit) and a memory 52 which is a storage device, as shown in Figure 25. For example, a program for performing the method in the embodiment may be stored in the memory 52, and each function may be realized by executing the program stored in the memory 52 with the processor 51.

[0160] The above program, when loaded into a computer, includes a set of instructions (or software code) for causing the computer to perform one or more of the functions described in the embodiments. The program may be stored on a non-temporary computer-readable medium or a physical storage medium. Examples, but not limited to, include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technologies, CD-ROM, digital versatile disc (DVD), Blu-ray® disc or other optical disc storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices. The program may be transmitted over a temporary computer-readable medium or a communication medium. Examples, but not limited to, include temporary computer-readable medium or a communication medium that includes electrically, optically, acoustically or otherwise propagating signals.

[0161] Although the present disclosure has been described above with reference to embodiments, the present disclosure is not limited to the embodiments described above. Various modifications to the structure and details of the present disclosure can be made as can be understood by those skilled in the art within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0162] Each drawing is merely illustrative to illustrate one or more embodiments. Each drawing may be associated with one or more other embodiments, rather than being associated with only one specific embodiment. As those skilled in the art will understand, various features or steps described with reference to any one drawing can be combined with features or steps shown in one or more other drawings, for example, to create embodiments not explicitly shown or described. Not all features or steps shown in any one drawing to illustrate an exemplary embodiment are necessarily required, and some features or steps may be omitted. The order of steps described in any of the drawings may be changed as appropriate.

[0163] Some or all of the above embodiments may also be described as follows, but are not limited to the following:

[0164] (Note 1) A data processing system comprising: a first interpolation unit that interpolates a first missing portion in the first depth coordinate data based on first depth coordinate data in a first frame and first reflectance data indicating the reflectance of each coordinate in the first depth coordinate data; and a motion compensation unit that performs motion compensation on the interpolated first depth coordinate data using a motion vector indicating the difference between the first depth coordinate data and the second depth coordinate data based on the interpolated first depth coordinate data and second depth coordinate data in a second frame. (Note 2) The data processing system according to Note 1, wherein the first interpolation unit interpolates the first missing portion in the first depth coordinate data using the depth around the first missing portion in the first depth coordinate data based on the reflectance around the first missing portion in the first reflectance data. (Note 3) The data processing system according to Note 1, wherein the first interpolation unit interpolates the first missing portion in the first depth coordinate data based on the first depth coordinate data and the first reflectance data using a first convolutional neural network. (Note 4) The data processing system according to any one of Notes 1 to 3, wherein the first interpolation unit interpolates the first missing portion in the first depth coordinate data based on first mask data indicating the first missing portion in the first depth coordinate data. (Note 5) The data processing system according to Note 4, further comprising a generation unit that generates the first mask data based on the first depth coordinate data and the first reflectance data. (Note 6) The data processing system according to any one of Notes 1 to 3, wherein the first interpolation unit interpolates the first missing portion in the first reflectance data based on the first reflectance data.(Note 7) A data processing system according to any one of Notes 1 to 3, comprising: a second interpolation unit that interpolates a second missing portion in the second depth coordinate data based on the second depth coordinate data in the second frame and second reflectance data indicating the reflectance of each coordinate in the second depth coordinate data; and a motion vector estimation unit that estimates the motion vector based on the interpolated first depth coordinate data and the interpolated second depth coordinate data. (Note 8) A data processing system according to Note 7, comprising: a compression unit that compresses the residual between the motion-compensated first depth coordinate data and the interpolated second depth coordinate data. (Note 9) A data processing system according to Note 8, comprising: an extension unit that extends the compressed residual and restores the second depth coordinate data based on the extended residual and the motion-compensated first depth coordinate data. (Note 10) A data processing system comprising: a first interpolation unit that interpolates a first missing portion in the first depth coordinate data based on first depth coordinate data in the first frame and first reflectance data indicating the reflectance of each coordinate in the first depth coordinate data; a second interpolation unit that interpolates a second missing portion in the second depth coordinate data based on second depth coordinate data in the second frame and second reflectance data indicating the reflectance of each coordinate in the second depth coordinate data; and a motion vector estimation unit that estimates a motion vector based on the interpolated first depth coordinate data and the interpolated second depth coordinate data. (Note 11) The data processing system according to Note 10, wherein the first interpolation unit interpolates the first missing portion in the first depth coordinate data using the depth around the first missing portion in the first depth coordinate data based on the reflectance around the first missing portion in the first reflectance data, and the second interpolation unit interpolates the second missing portion in the second depth coordinate data using the depth around the second missing portion in the second depth coordinate data based on the reflectance around the second missing portion in the second reflectance data.(Note 12) The data processing system according to Note 10, wherein the first interpolation unit interpolates the first missing portion in the first depth coordinate data based on the first depth coordinate data and the first reflectance data using a first convolutional neural network, and the second interpolation unit interpolates the second missing portion in the second depth coordinate data based on the second depth coordinate data and the second reflectance data using a second convolutional neural network. (Note 13) The data processing system according to any one of Notes 10 to 12, wherein the first interpolation unit interpolates the first missing portion in the first depth coordinate data based on first mask data indicating the first missing portion in the first depth coordinate data, and the second interpolation unit interpolates the second missing portion in the second depth coordinate data based on second mask data indicating the second missing portion in the second depth coordinate data. (Note 14) The data processing system according to Note 13, further comprising a generation unit that generates the second mask data based on the second depth coordinate data and the second reflectance data. (Note 15) The data processing system according to any one of Notes 10 to 12, wherein the first interpolation unit interpolates the first missing portion in the first reflectance data based on the first reflectance data, and the second interpolation unit interpolates the second missing portion in the second reflectance data based on the second reflectance data. (Note 16) A data processing method comprising: filling in a first missing portion in the first depth coordinate data based on first depth coordinate data in a first frame and first reflectance data showing the reflectance of each coordinate in the first depth coordinate data; and performing motion compensation on the filled first depth coordinate data using a motion vector showing the difference between the first depth coordinate data and the second depth coordinate data based on the filled first depth coordinate data and second depth coordinate data in a second frame.(Note 17) A data processing method comprising: filling in a first missing portion in the first depth coordinate data based on first depth coordinate data in the first frame and first reflectance data showing the reflectance of each coordinate in the first depth coordinate data; filling in a second missing portion in the second depth coordinate data based on second depth coordinate data in the second frame and second reflectance data showing the reflectance of each coordinate in the second depth coordinate data; and estimating a motion vector based on the filled-in first depth coordinate data and the filled-in second depth coordinate data. (Note 18) A program for causing a computer to perform a process that includes: filling in a first missing portion in the first depth coordinate data based on first depth coordinate data in a first frame and first reflectance data showing the reflectance of each coordinate in the first depth coordinate data; and performing motion compensation on the filled first depth coordinate data using a motion vector showing the difference between the first depth coordinate data and the second depth coordinate data based on the filled first depth coordinate data and second depth coordinate data in a second frame. (Note 19) A program for causing a computer to perform the following processes: filling in a first missing portion in the first depth coordinate data based on first depth coordinate data in the first frame and first reflectance data showing the reflectance of each coordinate in the first depth coordinate data; filling in a second missing portion in the second depth coordinate data based on second depth coordinate data in the second frame and second reflectance data showing the reflectance of each coordinate in the second depth coordinate data; and estimating a motion vector based on the completed first depth coordinate data and the completed second depth coordinate data.

[0165] Some or all of the elements (e.g., configurations and functions) described in Appendices 2-9 and 11-15, which are dependent on Appendices 1 and 10 (Data Processing Systems), may also be dependent on Appendices 16 and 17 (Data Processing Methods) and Appendices 18 and 19 (Programs) in the same way as in Appendices 2-9 and 11-15. Some or all of the elements described in any appendice may be applied to various hardware, software, recording means, systems, and methods for recording software.

[0166] This application claims priority based on Japanese Patent Application No. 2024-170284, filed on 30 September 2024, and incorporates all of its disclosures herein.

[0167] 1. Vehicle-infrastructure cooperation system 10, 20. Data processing system 11. First interpolation unit 12. Motion compensation unit 13. Second interpolation unit 14. Motion vector estimation unit 30, 40. Data processing device 50. Computer 51. Processor 52. Memory 100. Terminal 101. Lidar sensor 110. Acquisition unit 120. Encoder 130. Transmission unit 210. Receiving unit 220. Decoder 230. MEC processing unit 300. Base station 400. Center server 401. Sensor fusion function 402. Recognition function 403. Prediction function 900. Compression / decompression processing 901. Motion vector estimation / compression / decompression processing 902. Motion compensation processing 903. Residual compression / decompression processing 904. Missing data interpolation processing 1201. Current frame interpolation unit 1202. Previous frame interpolation unit 1203. Motion vector estimation unit 1204 Motion vector compression quantization unit 1205 Entropy coding unit 1206 Motion vector expansion unit 1207 Motion compensation unit 1208 Residual / reflectance compression quantization unit 1208a Residual / reflectance / mask compression quantization unit 1208b Residual / mask compression quantization unit 1209 Entropy coding unit 1210 Depth image expansion unit 1211 Frame storage unit 1212 Mask generation unit 2201 Entropy decoding unit 2202 Motion vector expansion unit 2203 Previous frame interpolation unit 2204 Motion compensation unit 2205 Entropy decoding unit 2206 Depth image expansion unit 2207 Reflectance expansion unit 2207a Reflectance / mask expansion unit 2207b Mask expansion unit 2208 Frame storage unit

Claims

1. A data processing system comprising: a first interpolation unit that interpolates a first missing portion in the first depth coordinate data based on first depth coordinate data in a first frame and first reflectance data indicating the reflectance of each coordinate in the first depth coordinate data; and a motion compensation unit that performs motion compensation on the interpolated first depth coordinate data using a motion vector indicating the difference between the first depth coordinate data and the second depth coordinate data based on the interpolated first depth coordinate data and second depth coordinate data in a second frame.

2. The data processing system according to claim 1, wherein the first interpolation unit interpolates the first missing portion in the first depth coordinate data using the depth around the first missing portion in the first depth coordinate data based on the reflectance around the first missing portion in the first reflectance data.

3. The data processing system according to claim 1, wherein the first interpolation unit interpolates the first missing portion in the first depth coordinate data based on the first depth coordinate data and the first reflectance data using a first convolutional neural network.

4. The data processing system according to any one of claims 1 to 3, wherein the first interpolation unit interpolates the first missing portion in the first depth coordinate data based on first mask data indicating the first missing portion in the first depth coordinate data.

5. The data processing system according to claim 4, further comprising a generation unit that generates the first mask data based on the first depth coordinate data and the first reflectance data.

6. The data processing system according to any one of claims 1 to 3, wherein the first interpolation unit interpolates the first missing portion in the first reflectance data based on the first reflectance data.

7. A data processing system according to any one of claims 1 to 3, comprising: a second interpolation unit that interpolates a second missing portion in the second depth coordinate data based on the second depth coordinate data in the second frame and second reflectance data indicating the reflectance of each coordinate of the second depth coordinate data; and a motion vector estimation unit that estimates the motion vector based on the interpolated first depth coordinate data and the interpolated second depth coordinate data.

8. The data processing system according to claim 7, further comprising a compression unit for compressing the residual between the first depth coordinate data after motion compensation and the second depth coordinate data after interpolation.

9. The data processing system according to claim 8, further comprising an extension unit that extends the compressed residual and restores the second depth coordinate data based on the extended residual and the first depth coordinate data on which motion compensation has been performed.

10. A data processing system comprising: a first interpolation unit that interpolates a first missing portion in the first depth coordinate data based on first depth coordinate data in a first frame and first reflectance data indicating the reflectance of each coordinate in the first depth coordinate data; a second interpolation unit that interpolates a second missing portion in the second depth coordinate data based on second depth coordinate data in a second frame and second reflectance data indicating the reflectance of each coordinate in the second depth coordinate data; and a motion vector estimation unit that estimates a motion vector based on the interpolated first depth coordinate data and the interpolated second depth coordinate data.

11. The data processing system according to claim 10, wherein the first interpolation unit interpolates the first missing portion in the first depth coordinate data using the depth around the first missing portion in the first depth coordinate data based on the reflectance around the first missing portion in the first reflectance data, and the second interpolation unit interpolates the second missing portion in the second depth coordinate data using the depth around the second missing portion in the second depth coordinate data based on the reflectance around the second missing portion in the second reflectance data.

12. The data processing system according to claim 10, wherein the first interpolation unit interpolates the first missing portion in the first depth coordinate data based on the first depth coordinate data and the first reflectance data using a first convolutional neural network, and the second interpolation unit interpolates the second missing portion in the second depth coordinate data based on the second depth coordinate data and the second reflectance data using a second convolutional neural network.

13. The data processing system according to any one of claims 10 to 12, wherein the first interpolation unit interpolates the first missing portion in the first depth coordinate data based on first mask data indicating the first missing portion in the first depth coordinate data, and the second interpolation unit interpolates the second missing portion in the second depth coordinate data based on second mask data indicating the second missing portion in the second depth coordinate data.

14. The data processing system according to claim 13, further comprising a generation unit that generates the second mask data based on the second depth coordinate data and the second reflectance data.

15. The data processing system according to any one of claims 10 to 12, wherein the first interpolation unit interpolates the first missing portion in the first reflectance data based on the first reflectance data, and the second interpolation unit interpolates the second missing portion in the second reflectance data based on the second reflectance data.

16. A data processing method comprising: filling in a first missing portion in the first depth coordinate data based on first depth coordinate data in a first frame and first reflectance data indicating the reflectance of each coordinate in the first depth coordinate data; and performing motion compensation on the filled-in first depth coordinate data using a motion vector indicating the difference between the first depth coordinate data and the second depth coordinate data based on the filled-in first depth coordinate data and second depth coordinate data in a second frame.

17. A data processing method comprising: filling in a first missing portion in the first depth coordinate data based on first depth coordinate data in the first frame and first reflectance data indicating the reflectance of each coordinate in the first depth coordinate data; filling in a second missing portion in the second depth coordinate data based on second depth coordinate data in the second frame and second reflectance data indicating the reflectance of each coordinate in the second depth coordinate data; and estimating a motion vector based on the filled-in first depth coordinate data and the filled-in second depth coordinate data.

18. A program for causing a computer to perform a process that includes: filling in a first missing portion in the first depth coordinate data based on first depth coordinate data in a first frame and first reflectance data showing the reflectance of each coordinate in the first depth coordinate data; and performing motion compensation on the filled first depth coordinate data using a motion vector showing the difference between the first depth coordinate data and the second depth coordinate data based on the filled first depth coordinate data and second depth coordinate data in a second frame.

19. A program for causing a computer to perform the following processes: filling in a first missing portion in the first depth coordinate data based on first depth coordinate data in the first frame and first reflectance data showing the reflectance of each coordinate in the first depth coordinate data; filling in a second missing portion in the second depth coordinate data based on second depth coordinate data in the second frame and second reflectance data showing the reflectance of each coordinate in the second depth coordinate data; and estimating a motion vector based on the completed first depth coordinate data and the completed second depth coordinate data.

Images