Encoding apparatus performing inter prediction operation based on an overlap frame, and operating method thereof

Abstract

According to an embodiment of the present disclosure, an encoder is disclosed that receives first to third input frames included within a first intra period and outputs a bit stream corresponding to a third input frame. The encoder according to an embodiment of the present disclosure may include a motion compensation unit that generates a first reference frame corresponding to a first input frame and a second reference frame corresponding to a second input frame, a union operation unit that generates an overlap frame by performing a union operation based on the first reference frame and the second reference frame, and an inter-predict unit that generates an occupancy code by performing an inter-prediction operation on the overlap frame and the third input frame. In this case, the bit stream may include an occupancy code.

Description

Technology Field

[0001] The present disclosure relates to an encoding device that encodes three-dimensional point cloud data and outputs a bit stream, and a method of operating the same. More specifically, the present disclosure relates to an encoding device that performs an inter-prediction operation based on an overlap frame and a method of operating the same. Background Technology

[0002] A point cloud can refer to a set of points existing in three-dimensional space. Point clouds can be applied in various fields, such as real-time 3D immersive telepresence, GIS (geographic information system), or navigation services.

[0003] However, point clouds have a larger capacity compared to 2D images because they contain actual spatial information about objects. Accordingly, various encoding and compression methods for processing point cloud data are being studied. The problem to be solved

[0004] The present disclosure is intended to solve the technical problem described above. More specifically, the object of the present disclosure is to provide an encoding device that performs an inter-prediction operation based on an overlap frame and a method of operating the same. means of solving the problem

[0005] According to an embodiment of the present disclosure, an encoder is disclosed that receives first to third input frames included within a first intra period and outputs a bit stream corresponding to the third input frame. The encoder comprises a motion compensation unit that generates a first reference frame corresponding to the first input frame and a second reference frame corresponding to the second input frame, a union operation unit that generates an overlap frame by performing a union operation based on the first reference frame and the second reference frame, and an inter-predict unit that generates an occupancy code by performing an inter-prediction operation on the overlap frame and the third input frame, wherein the bit stream may include the occupancy code.

[0006] In one embodiment, the first to third input frames may correspond to the first to third time points of point cloud data collected by a LiDAR device outside the encoder, respectively.

[0007] In one embodiment, the first input frame may include first detection points for objects, the second input frame may include second detection points for objects, and the third input frame may include third detection points for objects.

[0008] In one embodiment, the first reference frame may include first motion compensation detection points corresponding to the first detection points, and the second reference frame may include second motion compensation detection points corresponding to the second detection points.

[0009] In one embodiment, the overlap frame may include the first motion compensation detection points and the second motion compensation detection points.

[0010] In one embodiment, the first detection points are included in the first detection area of ​​the lidar device at the first time point, the second detection points are included in the second detection area of ​​the lidar device at the second time point, and the third detection points are included in the third detection area of ​​the lidar device at the third time point, and the overlap frame may include motion compensation detection points among the first motion compensation detection points and the second motion compensation detection points that are included within the third detection area.

[0011] In one embodiment, the overlap frame may not include motion compensation detection points among the first motion compensation detection points and the second motion compensation detection points that are not included within the third detection area.

[0012] In one embodiment, the union operation may be performed on any combination of a plurality of reference frames including the first reference frame and the second reference frame.

[0013] In one embodiment, the bit stream may include a bit region representing reference frames used in the union operation among the plurality of reference frames.

[0014] A method of operation of an encoder for encoding first to nth input frames to generate first to nth bit streams according to an embodiment of the present disclosure comprises: generating a first converted frame and a first bit stream based on the first input frame; and sequentially performing an encoding operation for the second to nth input frames to generate the second to nth bit streams, respectively, wherein the encoding operation for the k-th input frame among the second to nth input frames comprises: performing a motion prediction operation for the k-th input frame and the k-1 converted frame to generate a k-1 motion vector; generating first to k-1 reference frames for the k-th input frame based on the first to k-1 motion vectors; performing a union operation for the first to k-1 reference frames for the k-th input frame to generate a k-1 overlap frame; performing an inter-prediction operation for the k-1 overlap frame and the k-th input frame to generate a k-th occupancy code; and the k-th occupancy code and the It may include the step of generating a k-th transformed frame based on a k-1st overlap frame, and the step of generating a k-th bit stream based on the k-th occupancy code and the k-1st motion vector.

[0015] In one embodiment, the first to nth input frames may be included in the same intra-period.

[0016] In one embodiment, the first to nth input frames may each include detection points within different detection regions.

[0017] In one embodiment, the first to k-1 reference frames for the k-th input frame may be the result of performing a motion compensation operation on each of the first to k-1 transformed frames based on the k-th input frame, based on the first to k-1 motion vectors.

[0018] In one embodiment, the k-1 overlap frame may include motion-compensated detection points included in each of the first to k-1 reference frames for the k-th input frame.

[0019] In one embodiment, the k-th bit stream may include a first bit area representing an encoding method, a second bit area representing the k-1st motion vector, and a third bit area representing the k-th occupancy code.

[0020] A method of operation of a decoder for decoding first to n-th bit streams to generate first to n-th decoded frames according to an embodiment of the present disclosure comprises: generating a first decoded frame based on a first occupying code included in a first bit stream; and sequentially performing a decoding operation for the second to n-th bit streams to generate the second to n-th decoded frames, respectively, wherein the decoding operation for the k-th bit stream among the second to n-th bit streams may include: generating first to k-1 reference frames for the k-th bit stream based on first to k-1 motion vectors; generating a k-th interprediction frame based on the first to k-1 reference frames for the k-th bit stream; and generating a k-th decoded frame based on the k-th interprediction frame and the k-th occupying code.

[0021] In one embodiment, the k-th interprediction frame may be an overlap frame generated through a union operation of the first to k-1 reference frames.

[0022] In one embodiment, the k-th bit stream may include a first bit area representing an encoding method, a second bit area representing the k-1 motion vector, and a third bit area representing the k-th occupancy code.

[0023] In one embodiment, the k-th interprediction frame may be selected from a k-th interprediction frame group comprising a plurality of overlap frames each generated through a union operation for any combination of the first to k-1 reference frames.

[0024] In one embodiment, the k-th bit stream may include a first bit area representing an encoding method, a second bit area representing the k-1 motion vector, a third bit area representing the k-th occupancy code, and a fourth bit area representing reference frames among the first to k-1 reference frames used to generate the k-th interprediction frame. Effects of the invention

[0025] According to an embodiment of the present disclosure, an encoding device and a method of operating the same are provided for performing an inter-prediction operation based on an overlap frame. More specifically, the overlap frame may be generated through a combination of frames prior to the target frame on which the inter-prediction operation is to be performed. In this case, since more detection points may be considered during inter-prediction for the target frame, an encoding device and a method of operating the same with improved compression performance may be provided. Brief explanation of the drawing

[0026] FIG. 1 is a block diagram showing an encoding device according to an embodiment of the present disclosure. Figure 2 is a diagram showing multiple input frames of Figure 1. FIG. 3 is a drawing showing the encoder (100) of FIG. 1 in more detail. FIG. 4 shows the operation of the encoder of FIG. 3 according to one embodiment. FIG. 5 shows a method of operation of the encoding device of FIG. 1 according to an embodiment of FIG. 4. FIG. 6 is a table showing the target frame and the inter-prediction frame according to the embodiment of FIG. 4. FIG. 7 shows the operation of the encoder of FIG. 3 according to an embodiment of the present disclosure. FIG. 8 shows a method of operation of the encoding device of FIG. 1 according to an embodiment of FIG. 7. FIG. 9 shows an example of step S140 of FIG. 8. FIG. 10 is a drawing showing steps S142 to S144 of FIG. 9. Figure 11 is a diagram showing the operation of the union operation unit of Figure 3. FIG. 12 is a table showing a target frame and an inter-prediction frame according to an embodiment of FIG. 9. FIG. 13 shows the operation of the encoder of FIG. 3 according to another embodiment of the present disclosure. FIG. 14 shows another embodiment of step S140 of FIG. 8 according to the embodiment of FIG. 13. FIG. 15 is a drawing showing steps S242 to S245 of FIG. 14. FIG. 16 is a table showing a target frame and an inter-prediction frame group according to an embodiment of FIG. 13. FIG. 17 shows the configuration of a bit stream according to an embodiment of the present disclosure. FIG. 18 shows a decoder according to an embodiment of the present disclosure. FIG. 19 shows the operation of the decoder of FIG. 18 according to an embodiment of the present disclosure. FIG. 20 shows a method of operation of the decoding device of FIG. 18 according to an embodiment of FIG. 19. Figure 21 shows step S330 of Figure 20 in more detail. FIG. 22 shows the operation of the decoder of FIG. 18 according to another embodiment of the present disclosure. FIG. 23 shows another embodiment of step S330 of FIG. 20 according to the embodiment of FIG. 22. FIG. 24 is a block diagram showing a lidar system according to an embodiment of the present disclosure. Specific details for implementing the invention

[0027] In the following, embodiments of the present disclosure will be described clearly and in detail so that a person skilled in the art can easily practice the present disclosure. Detailed matters, such as detailed configurations and structures, are provided merely to aid in the overall understanding of the embodiments of the present disclosure. Therefore, variations of the embodiments described herein may be made by those skilled in the art without departing from the technical spirit and scope of the present disclosure. Furthermore, descriptions of well-known functions and structures are omitted for the sake of clarity and brevity. Configurations in the drawings or detailed description below may be connected to other components other than those shown in the drawings or described in the detailed description. Terms used in the text are defined with respect to the functions of the present disclosure and are not limited to specific functions. The definitions of terms may be determined based on the details described in the detailed description.

[0028] Components described by reference to terms such as driver or block used in the detailed description may be implemented in the form of software, hardware, or a combination thereof. For example, software may be machine code, firmware, embedded code, and application software. For example, hardware may include electrical circuits, electronic circuits, processors, computers, integrated circuit cores, pressure sensors, inertial sensors, Micro Electro Mechanical Systems (MEMS), passive components, or a combination thereof.

[0029] FIG. 1 is a block diagram showing an encoding device according to an embodiment of the present disclosure. Referring to FIG. 1, the encoding device (ED) may include a transform unit (TF) and an encoder (100). The encoding device (ED) may encode point cloud data (PC) and output a bitstream (BS). The operation of each component of the encoding device (ED) is described in detail below.

[0030] The conversion unit (TF) may receive point cloud data (PC) from an external device (not shown). The point cloud data (PC) may include information about points existing in three-dimensional space. For example, the point cloud data (PC) may include x-axis coordinate, y-axis coordinate, and z-axis coordinate information of an object existing in three-dimensional space. However, the scope of the present disclosure is not limited thereto, and the point cloud data (PC) may include various forms of coordinate information such as Cartesian coordinates, cylindrical coordinates, spherical coordinates, etc.

[0031] In one embodiment, point cloud data (PC) may be collected from various types of external devices (not shown), such as a LiDAR device, an RGB-D device, a 3D scanner device, etc. For a more concise explanation, an embodiment in which point cloud data (PC) is collected by a LiDAR device will be described as representative below. However, the scope of the present disclosure is not limited thereto.

[0032] The transformation unit (TF) can transform point cloud data (PC) to output multiple input frames (IFM). For example, the transformation unit (TF) can perform operations such as scaling, translation, rotation, and skew on the point cloud data (PC). The multiple input frames (IFM) may be in a form suitable for processing by internal components of an encoding device (ED).

[0033] In one embodiment, a plurality of input frames (IFM) may be referred to as internal frames, raw frames, or internal format frames.

[0034] In one embodiment, each of the plurality of input frames (IFM) may include detection points. For example, the detection points may include geometric information (e.g., physical coordinate information) about an object detected by an external device. However, the scope of the present disclosure is not limited thereto, and the detection points may include various attribute information such as reflection intensity information, color information, transparency information, normal vector information, and material information of the object. However, an embodiment in which the detection points represent geometric information of the object (e.g., x, y, z coordinate information) will be described as representative below.

[0035] In one embodiment, each of the plurality of input frames (IFM) may correspond to point cloud data (PC) at different points in time. That is, each of the plurality of input frames (IFM) may correspond to information regarding different points in time of the point cloud data (PC). For example, each of the plurality of frames (FM) may have a chronological order.

[0036] In one embodiment, point cloud data (PC) may be collected based on a moving LiDAR device. That is, the LiDAR device can detect objects included in a moving detection area. In this case, the center of the detection area (e.g., the physical location of the LiDAR device) may be referred to as the detection bull.

[0037] In one embodiment, each of the plurality of input frames (IFM) may be included in the same intra period (or intra frame period).

[0038] The encoder (100) can receive a plurality of input frames (IFM) and generate a plurality of bit streams (BS). For example, the encoder (100) can sequentially encode a plurality of input frames (IFM) to sequentially generate a plurality of bit streams (BS). However, the scope of the present disclosure is not limited thereto, and the encoder (100) can perform an encoding operation on a plurality of input frames (IFM) in an order different from the input order. For example, among a plurality of input frames (IFM), a fourth input frame may be encoded before a third input frame that is input before the fourth input frame. However, for the sake of brevity, an example in which a plurality of input frames (IFM) are sequentially encoded will be described as representative below. The specific configuration and operation of the encoder (100) will be described in more detail with reference to the drawings below.

[0039] FIG. 2 is a diagram showing a plurality of input frames of FIG. 1. Referring to FIG. 1 and FIG. 2, the conversion unit (TF) can convert point cloud data (PC) and output a plurality of input frames (IFM).

[0040] Multiple input frames (IFMs) may have a chronological order. For example, the multiple input frames (IFMs) may include first to nth input frames (IFM1 to IFMn). In this case, the first to nth input frames (IFM1 to IFMn) may correspond to the information of the first to nth time points (t1 to tn) of the point cloud data (PC), respectively.

[0041] The first to nth input frames (IFM1~IFMn) may be included within the same intra period. For example, the first to nth input frames (IFM1~IFMn) may be included in the first intra period (intra period #1).

[0042] In one embodiment, an intra-cycle may refer to a unit in which an encoding operation is performed. For example, an encoding operation for a target frame having the earliest order among frames included within a single intra-cycle (e.g., a first input frame within a first intra-cycle) may be performed without considering input frames preceding the target frame.

[0043] For a more concise explanation, the encoding operation for the first to nth input frames (IFM1 to IFMn) included within a single intra period will be described as a representative example below. However, the scope of this disclosure is not limited thereto, and a second intra period other than the first intra period may also be included within the plurality of input frames (IFM). For example, the plurality of input frames (IFM) may also include an n+1th input frame included in the second intra period.

[0044] FIG. 3 is a drawing showing the encoder (100) of FIG. 1 in more detail. Referring to FIG. 1 and FIG. 3, the encoder (100) may include a motion estimation unit (110), a motion compensation unit (120), a union operation unit (130), an inter predict unit (140), a reconstruct unit (150), and a memory unit (160).

[0045] The encoder (100) can sequentially perform encoding operations on the first to nth input frames (IFM1~IFMn). For a more concise explanation, the input frame among the first to nth input frames (IFM1~IFMn) on which the encoding operation is performed will be referred to as the target frame.

[0046] The motion prediction unit (110) can perform motion prediction operations for two different frames. That is, the motion prediction unit (110) can calculate a motion vector between two different frames. For example, the motion prediction unit (110) can calculate a first motion vector between a first frame and a second frame.

[0047] In one embodiment, the motion prediction unit (110) can perform a motion prediction operation based on an iterative closest point (ICP) algorithm.

[0048] The motion compensation unit (120) can compensate for motion between two different frames based on a motion vector. For example, the motion compensation unit (120) can generate a reference frame for the target frame by compensating for motion for any frame based on the target frame. For example, the motion compensation unit (120) can generate a first reference frame for the second frame by multiplying the coordinates of detection points included in the first frame by a first motion vector. In this case, the first reference frame for the second frame may refer to the first frame that has been motion-compensated based on the second frame.

[0049] In one embodiment, the motion prediction operation and the motion compensation operation may refer to operations that predict and compensate for the movement (e.g., translation, rotation, etc.) of a device (e.g., LiDAR device) that detects detection points for a target frame and another frame (for a more concise explanation, the other frame may be referred to as a comparison frame). For example, the motion prediction operation may be performed by calculating the movement (e.g., translation, rotation) between the geometric information of an object (e.g., detection points) included in the target frame and the geometric information of an object included in the comparison frame. More specifically, the motion prediction operation may refer to an operation that calculates a motion vector between the target frame and the comparison frame. Meanwhile, the motion compensation operation may refer to an operation that corrects the geometric information of objects (e.g., detection points) included in the comparison frame using the motion vector calculated through the motion prediction operation. In this case, the 'corrected comparison frame' may be referred to as the 'reference frame', and the 'detection points included in the reference frame' may be referred to as 'motion-compensated detection points'. However, the scope of this disclosure is not limited to the terms described above.

[0050] The union operation unit (130) can generate an overlap frame based on any combination of reference frames. For example, the union operation unit (130) can generate an overlap frame by performing a union operation on two or more of the reference frames. For example, the overlap frame can be generated through a union (or may also be referred to as a union operation) operation on detection points included in different reference frames. The operation of the overlap frame and the union operation unit (130) is described in more detail with reference to FIG. 10 below.

[0051] The inter-predict section (140) can generate an occupancy code by performing an inter-predict operation on an inter-predict frame and a target frame. In this case, the inter-predict operation may be an operation that compares the inter-predict frame and the target frame.

[0052] In one embodiment, the inter-prediction frame may refer to a frame used to generate a possession code through comparison with a target frame. For example, the inter-prediction frame may be one of the reference frames or one of the overlap frames.

[0053] In one embodiment, when the inter-prediction frame is an overlap frame, the efficiency of the inter-prediction operation for the target frame may be increased. More specifically, the compression efficiency for the target frame may be increased. For example, the overlap frame may contain more detection points than the reference frame. Therefore, when the inter-prediction operation is performed on the overlap frame and the target frame, more detection points can be considered, and thus the compression efficiency for the target frame may be improved.

[0054] In one embodiment, the inter-predict unit (140) can perform an inter-predict operation based on the octree structure of the inter-predict frame and the target frame. For example, the inter-predict unit (140) can generate a binary code for the inter-predict frame and the target frame based on the octree structure. The inter-predict unit (140) can analyze the occupancy bit pattern of the binary code for the target frame based on the binary code for the inter-predict frame. More specifically, the inter-predict unit (140) can utilize the binary code for the inter-predict frame and the target frame (FMk) as an input value for an arithmetic coding context. In this case, an occupancy code associated with the target frame can be generated.

[0055] The reconstruction unit (150) can generate a first converted frame by performing a quantization operation on the first input frame (IFM1). The reconstruction unit (150) can generate a converted frame corresponding to the input frame based on a reference frame and a occupied code. For a more concise explanation, the converted frame is referred to below using the reference symbol 'CFM'.

[0056] In one embodiment, the converted frame (CFM) may refer to a frame that is restored when a decoding operation is performed based on a bit stream (BS). For example, when a decoding operation is performed on a first bit stream (BS1), a first converted frame (CFM1) may be restored, and when a decoding operation is performed on an n-th bit stream (BSn), an n-th converted frame (CFMn) may be restored. However, the scope of the present disclosure is not limited thereto.

[0057] The memory unit (160) can store information generated during the performance of encoding operations for the first to nth input frames (IFM1~IFMn). For example, the memory unit (160) can temporarily store various information such as motion vectors, converted frames, and overlap frames.

[0058] The encoder (100) can perform an encoding operation on the first to nth input frames (IFM1 to IFMn) to generate the first to nth bit streams (BS1 to BSn). In this case, the first bit stream (BS1) may include a first occupancy code for the first input frame (IFM1). And, each of the second to nth bit streams (BS2 to BSn) may include an occupancy code and a motion vector for the corresponding input frame.

[0059] FIG. 4 shows the operation of the encoder of FIG. 3 according to one embodiment. With reference to FIG. 1 to FIG. 4, the encoding operation for the second to nth input frames (IFM2 to IFMn) will be described. For example, FIG. 4 is representatively described as an embodiment in which the kth input frame (IFMk) (i.e., k may be an integer greater than 2) among the second to nth input frames (IFM2 to IFMn) is the target frame. Since the encoding operation for the first input frame (IFM1) is similar to that described above with reference to FIG. 3, a detailed description is omitted.

[0060] Referring further to FIG. 4, the encoder (100) can receive the k-th input frame (IFMk). The motion prediction unit (110) can perform a motion prediction operation based on the k-th input frame (IFMk) and the k-1st transformed frame (CFMk-1). That is, the motion prediction unit (110) can generate the k-1st motion vector (MVk-1) based on the k-th input frame (IFMk) and the k-1st transformed frame (CFMk-1).

[0061] The motion compensation unit (120) can perform a motion compensation operation for the k-1 transformed frame (CFMk-1) based on the k-1 motion vector (MVk-1). That is, the motion compensation unit (120) can perform a motion compensation operation for the k-1 transformed frame (CFMk-1) based on the k-1 input frame (IFMk). For example, the motion compensation unit (120) can generate the k-1 reference frame (RFMk-1) by multiplying the coordinates of the detection points of the k-1 transformed frame (CFMk-1) by the k-1 motion vector (MVk-1).

[0062] The inter-predict section (140) can generate the k-th occupy code (OCk) by performing an inter-predict operation on the k-th reference frame (RFMk-1) and the k-th input frame (IFMk).

[0063] The reconstruction unit (150) can generate a k-th converted frame (CFMk) based on the k-1st reference frame (RFMk-1) and the k-th occupied code (OCk). The k-th converted frame (CFMk) may correspond to the k-th input frame (IFMk). For example, when a decoding operation is performed based on the k-th bit stream (BSk), the k-th converted frame (CFMk) can be restored. The decoding operation is described in more detail with reference to FIGS. 18 through 23.

[0064] The encoder (100) can generate a k-th bit stream (BSk) based on a k-th input frame (IFMk). In this case, the k-th bit stream (BSk) may include a k-th occupy code (OCk) and a k-th motion vector (MVk-1).

[0065] FIG. 5 shows a method of operation of the encoding device of FIG. 1 according to an embodiment of FIG. 4. Referring to FIG. 1 to FIG. 5, in step S11, the encoding device (ED) can generate first to nth input frames (IFM1 to IFMn) based on point cloud data (PD).

[0066] In step S12, the encoding device (ED) can generate a first bit stream (BS1) and a first converted frame (CFM1) based on a first input frame (IFM1). For example, the encoder (100) can generate a first converted frame (CFM1) based on the first input frame (IFM1) through a reconstruction unit (150). The encoder (100) can generate a first occupy code (OC1) for the first input frame (IFM1). The encoder (100) can generate a first bit stream (BS1) based on the first occupy code (OC1). In this case, the first bit stream (BS1) may include the first occupy code (OC1).

[0067] In step S13, the variable (k) may be set to 2. The variable (k) is used to describe that the encoding operation is performed sequentially for the second to nth input frames (IFM2~IFMn) and does not limit the scope of the present disclosure.

[0068] In step S14, the encoding device (ED) can generate the k-1 motion vector (MVk-1) by performing a motion prediction operation on the k-th input frame (IFMk) and the k-1 transformed frame (CFMk-1). For example, the motion prediction unit (110) can calculate the k-1 motion vector (MVk-1) between the k-th input frame (IFMk) and the k-1 transformed frame (CFMk-1).

[0069] In one embodiment, the k-1 motion vector (MVk-1) can be stored in the memory unit (160).

[0070] In step S15, the encoding device (ED) may generate the k-1 reference frame (RFMk-1) by applying the k-1 motion vector (MVk-1) to the k-1 converted frame (CFMk-1). For example, the motion compensation unit (120) may generate the k-1 reference frame (RFMk-1) by multiplying the k-1 motion vector (MVk-1) by the coordinates of the detection points included in the k-1 converted frame (CFMk-1). In this case, the k-1 reference frame (RFMk-1) may be a motion-compensated frame based on the k-th input frame (IFMk) (i.e., the target frame).

[0071] In step S16, the encoding device (ED) can generate the k-th occupy code (OCk) by performing an inter-predict operation on the k-th reference frame (RFMk-1) and the k-th input frame. For example, the inter-predict unit (140) can generate the k-th occupy code (OCk) by performing an inter-predict operation on the k-th reference frame (RFMk-1) and the k-th input frame.

[0072] In step S17, the encoding device (ED) can generate the k-th transformed frame (CFMk) based on the k-th occupied code (OCk) and the k-1st reference frame (RFMk-1). For example, the encoder (100) can generate the k-th transformed frame (CFMk) based on the k-th occupied code (OCk) and the k-1st reference frame (RFMk-1) through the reconstruction unit (150). In this case, the k-th transformed frame (CFMk) can be used when an encoding operation is performed on the k+1st input frame (IFMk+1).

[0073] In one embodiment, the k-th converted frame (CFMk) can be stored in the memory unit (160).

[0074] In step S18, the encoding device (ED) can generate a k-th bit stream (BSk) based on a k-th occupy code (OCk) and a k-th motion vector (MVk-1). For example, the encoder (100) can generate a k-th bit stream (BSk) containing a k-th occupy code (OCk) and a k-th motion vector (MVk-1).

[0075] In step S19, it is determined whether the variable (k) value is n. If the variable (k) value is not n, step S20 may be performed, and the variable (k) may be increased by 1. If the variable (k) value is n, the operation of the encoding device (100) may be terminated. That is, in step S19, it can be determined whether all input frames included within the same intra-cycle have been encoded. In step S20, the target frame may be changed from the k-th input frame (IFMk) to the k+1-th input frame (IFMk+1).

[0076] FIG. 6 is a table showing a target frame and an inter-prediction frame according to an embodiment of FIG. 4. Referring to FIG. 3 through 6, the inter-prediction unit (140) can perform an inter-prediction operation based on the target frame and the inter-prediction frame.

[0077] The inter-predict unit (140) can perform an inter-predict operation based on the k-1 reference frame (RFMk-1) when the k-th input frame (IFMk) is a target frame. For example, the inter-predict unit (140) can perform an inter-predict operation based on the first reference frame (RFM1) when the second input frame (IFM2) is a target frame.

[0078] FIG. 7 illustrates the operation of the encoder of FIG. 3 according to an embodiment of the present disclosure. With reference to FIG. 1 through 3 and FIG. 7, the encoding operation for the second to nth input frames (IFM2 to IFMn) will be described. For example, FIG. 7 is representatively described for the case where the kth input frame (IFMk) (i.e., k may be an integer greater than 2) among the second to nth input frames (IFM2 to IFMn) is the target frame. The encoding operation for the first input frame (IFM1) is similar to that described above with reference to FIG. 3, so a detailed description is omitted.

[0079] Referring further to FIG. 7, the encoder (100) can receive the k-th input frame (IFMk). The motion prediction unit (110) can perform a motion prediction operation based on the k-th input frame (IFMk) and the k-1st transformed frame (CFMk-1). That is, the motion prediction unit (110) can generate the k-1st motion vector (MVk-1) based on the k-th input frame (IFMk) and the k-1st transformed frame (CFMk-1).

[0080] The motion compensation unit (120) can perform motion compensation operations for the first to k-1 transformed frames (CFM1 to CFMk-1) based on the first to k-1 motion vectors (MV1 to MVK-1). That is, the motion compensation unit (120) can perform motion compensation operations for the first to k-1 transformed frames (CFM1 to CFMk-1) based on the k-th input frame (IFMk). For example, the motion compensation unit (120) can generate a 'k-1 reference frame for the k-th input frame' (RFMk-1_k) by multiplying the coordinates of the detection points of the k-1 transformed frame (CFMk-1) by the k-1 motion vector (MVK-1). The motion compensation unit (120) can generate a ‘k-2nd reference frame for the k-th input frame’ (RFMk-2_k) by multiplying the coordinates of the detection points of the k-2nd transformed frame (CFMk-2) by the k-1st motion vector (MVk-1) and the k-2nd motion vector (MVk-2). Similarly, the motion compensation unit (120) can generate a ‘i-th reference frame for the k-th input frame’ (RFMi_k) by multiplying the coordinates of the detection points of the i-th transformed frame (CFMi) (where i is an integer greater than or equal to 1 and less than k) by the i-th to k-1st motion vectors (MVi~MVk-1).

[0081] The union operation unit (130) can generate the k-1st overlap frame (OFMk-1) by performing a union operation on the 'first reference frame for the k-th input frame' (RFM1_k) to the 'k-1st reference frame for the k-th input frame' (RFMk-1_k).

[0082] In one embodiment, the k-1st overlap frame (OFMk-1) may include all of the sensing points (i.e., motion-compensated sensing points) included in the 'first reference frame for the k-th input frame' (RFM1_k) to the 'k-1st reference frame for the k-th input frame' (RFMk-1_k). However, the scope of the present disclosure is not limited thereto.

[0083] The inter-predict section (140) can generate the k-th occupy code (OCk) by performing an inter-predict operation on the k-th overlap frame (OFMk-1) and the k-th input frame (IFMk).

[0084] The reconstruction unit (150) can generate a k-th converted frame (CFMk) based on the k-th overlap frame (OFMk-1) and the k-th occupy code (OCk). The k-th converted frame (CFMk) can correspond to the k-th input frame (IFMk). For example, when a decoding operation is performed based on the k-th bit stream (BSk), the k-th converted frame (CFMk) can be restored.

[0085] The encoder (100) can generate a k-th bit stream (BSk) based on a k-th input frame (IFMk). In this case, the k-th bit stream (BSk) may include a k-th occupy code (OCk) and a k-1-th motion vector (MVk-1).

[0086] That is, according to an embodiment of the present disclosure, an inter-prediction frame for the k-th input frame (IFMk) can be determined as the k-1st overlap frame (OFMk-1). In this case, since more detection points can be considered in the inter-prediction operation, the encoding efficiency of the encoder (100) can be increased.

[0087] FIG. 8 shows a method of operation of the encoding device of FIG. 1 according to an embodiment of FIG. 7. Referring to FIG. 1 to 3 and FIG. 7 to 8, in step S110, the encoding device (ED) can generate first to nth input frames (IFM1 to IFMn) based on point cloud data (PD).

[0088] In step S120, the encoding device (ED) can generate a first bit stream (BS1) and a first converted frame (CFM1) based on the first input frame (IFM1). Since step S120 is substantially the same as step S12 described earlier, a detailed description is omitted.

[0089] In step S130, the variable (k) may be set to 2. The variable (k) is used to describe that the encoding operation is performed sequentially for the second to nth input frames (IFM2~IFMn) and does not limit the scope of the present disclosure.

[0090] In step S140, the encoding device (ED) can perform an encoding operation on the k-th input frame (IFMk) to generate the k-th converted frame (CFMk) and the k-th bit stream (BSk). That is, in step S140, the encoder (100) can perform an encoding operation on the k-th input frame (IFMk). The operation of the encoder (100) in step S140 is described in more detail with reference to FIG. 9 below.

[0091] In step S150, it is determined whether the variable (k) value is n. If the variable (k) value is not n, step S160 may be performed, and the variable (k) may be increased by 1. If the variable (k) value is n, the operation of the encoding device (100) may be terminated. That is, in step S150, it can be determined whether all input frames included within the same intra-cycle have been encoded. In step S160, the target frame may be changed from the k-th input frame (IFMk) to the k+1-th input frame (IFMk+1).

[0092] FIG. 9 shows an example of step S140 of FIG. 8. Referring to FIG. 1 through 3 and FIG. 7 through 9, step S140 may include the following steps S141 through S146.

[0093] In step S141, the encoder (100) can generate a k-1 motion vector (MVk-1) by performing a motion prediction operation on the k-th input frame (IFMk) and the k-1 transformed frame (CFMk-1). For example, the motion prediction unit (110) can calculate a k-1 motion vector (MVk-1) between the k-th input frame (IFMk) and the k-1 transformed frame (CFMk-1).

[0094] In one embodiment, the k-1st motion vector (MVk-1) may be stored in the memory unit (160). For example, if the k-th input frame (IFMk) is a target frame, the memory unit (160) may store the first motion vector (MV1) to the k-1st motion vector (MVk-1).

[0095] In step S142, the encoder (100) can generate a ‘first reference frame for the k-th input frame’ (RFM1_k) to a ‘k-1 reference frame for the k-th input frame’ (RFMk-1_k) based on the first to k-1 motion vectors (MV1 to MVk-1). For example, the motion compensation unit (120) can generate a ‘i-th reference frame for the k-th input frame’ (RFMi_k) by multiplying the coordinates of the detection points of the i-th transformed frame (CFMi) (where i is an integer greater than or equal to 1 and less than k) by the i-th to k-1 motion vectors (MVi to MVk-1). In this case, the ‘i-th reference frame for the k-th input frame’ (RFMi_k) may refer to the result of motion compensating the i-th transformed frame (CFMi) corresponding to the i-th input frame (IFMi) based on the k-th input frame (IFMk).

[0096] In step S143, the encoder (100) can generate the k-1st overlap frame (OFMk-1) by performing a union operation on the 'first reference frame for the k-th input frame' (RFM1_k) to the 'k-1st reference frame for the k-th input frame' (RFMk-1_k). For example, the union operation unit (130) can generate the k-1st overlap frame (OFMk-1) through a union operation on the detection points (i.e., motion-compensated detection points) included in the 'first reference frame for the k-th input frame' (RFM1_k) to the 'k-1st reference frame for the k-th input frame' (RFMk-1_k).

[0097] In step S144, the encoder (100) can generate the k-th occupied code (OCk) by performing an inter-prediction operation on the k-th overlap frame (OFMk-1) and the k-th input frame (IFMk). For example, the inter-predict unit (140) can generate the k-th occupied code (OCk) by performing an inter-prediction operation on the k-th overlap frame (OFMk-1) and the k-th input frame (IFMk).

[0098] In step S145, the encoder (100) can generate the k-th transformed frame (CFMk) based on the k-th overlap frame (OFMk-1) and the k-th occupying code (OCk). For example, the reconstruction unit (150) can generate the k-th transformed frame (CFMk) based on the k-th overlap frame (OFMk-1) and the k-th occupying code (OCk).

[0099] In one embodiment, the k-th converted frame (CFMk) may be stored in the memory unit (160). For example, if the k-th input frame (IFMk) is a target frame, the memory unit (160) may store the first to k-th converted frames (CFM1~CFMk). The first to k-th converted frames (CFM1~CFMk) stored in the memory unit (160) may be used for an encoding operation for the k+1-th input frame (IFMk+1).

[0100] In step S146, the encoder (100) may generate a k-th bit stream (BSk) based on a k-th occupancy code (OCk) and a k-1-th motion vector (MVk-1). That is, the k-th bit stream (BSk) may include a k-th occupancy code (OCk) and a k-1-th motion vector (MVk-1). However, the scope of the present disclosure is not limited thereto, and the k-th bit stream (BSk) may include a wider variety of information. The configuration of the k-th bit stream (BSk) is described in more detail below with reference to FIG. 17.

[0101] FIG. 10 is a diagram showing steps S142 through S144 of FIG. 9. Referring to FIGS. 1 through 3 and FIGS. 7 through 10, the k-1st overlap frame (OFMk-1) can be generated through a union operation for the 'first reference frame for the k-th input frame' (RFM1_k) to the 'k-1st reference frame for the k-th input frame' (RFMk-1_k).

[0102] The k-1st overlap frame (OFMk-1) can be used for inter-predict operations with the k-th input frame (IMFk) (i.e., the target frame). That is, the k-1st overlap frame (OFMk-1) can be used as an inter-predict frame for the k-th input frame (IMFk).

[0103] Each of the ‘first reference frame for the k-th input frame’ (RFM1_k) to the ‘k-1 reference frame for the k-th input frame’ (RFMk-1_k) can be generated by motion compensating the detection points of the first to k-th transformed frames (CFM1~CFMk) based on the k-th input frame (IFMk).

[0104] The k-1st overlap frame (OFMk-1) may include detection points of the 'first reference frame for the k-th input frame' (RFM1_k) to the 'k-1st reference frame for the k-th input frame' (RFMk-1_k). Accordingly, the k-1st overlap frame (OFMk-1) may include more detection points than each of the detection points of the 'first reference frame for the k-th input frame' (RFM1_k) to the 'k-1st reference frame for the k-th input frame' (RFMk-1_k). Therefore, according to an embodiment of the present disclosure, more detection points can be considered in the inter-predict operation, so encoding efficiency can be improved.

[0105] FIG. 11 is a diagram showing the operation of the union operation unit of FIG. 3. Hereinafter, with reference to FIGS. 1 to 3 and FIGS. 7 to 11, an embodiment in which a third input frame (IFM3) is a target frame (i.e., k is 3) and a second overlap frame (OFM2) is generated through a union operation of a 'first reference frame for the third input frame' (RFM1_3) and a 'second reference frame for the third input frame' (RFM2_3) will be described as representative. However, the scope of the present disclosure is not limited thereto. For example, according to an embodiment of the present disclosure, an overlap frame may be generated through a union operation of two or more reference frames. An embodiment in which an overlap frame is generated through a union operation of two or more reference frames is described below with reference to FIGS. 13 to 16.

[0106] Referring further to FIG. 11, the 'first reference frame for the third input frame' (RFM1_3) may correspond to the first input frame (IFM1). For example, the 'first reference frame for the third input frame' (RFM1_3) may include detection points corresponding to the first time point (t1) information of the point cloud data (PC), just like the first input frame (IFM1). Similarly, the 'second reference frame for the third input frame' (RFM2_3) may correspond to the second input frame (IFM2) and may include detection points corresponding to the second time point (t2) information of the point cloud data (PC).

[0107] In one embodiment, the lidar device may move in a time-series manner. The lidar device may move on an operating area (OS). In this case, the detection bull corresponding to the physical location of the lidar device may also move in a time-series manner. For example, the lidar device may be placed at a first detection bull (DB1) at a first time point (t1), at a second detection bull (DB2) at a second time point (t2), and at a third detection bull (DB3) at a third time point (t3).

[0108] In one embodiment, an area within a specific distance relative to the detection center may be set as the detection area of ​​the LiDAR device. In this case, the detection area may move chronologically on the operating area (OS). For example, the first detection area (DA1) at the first time point (t1) may be located at a position 'L1' away in the second direction from the starting point (PA) on the operating area (OS). The second detection area (DA2) at the second time point (t2) may be located at a position 'L2' away in the second direction from the starting point (PA) on the operating area (OS). And the third detection area (DA3) at the third time point (t3) may be located at a position 'L3' away in the second direction from the starting point (PA) on the operating area (OS).

[0109] A first object (OBJ1) and a second object (OBJ2) may be included on the operating area (OS). The LiDAR device may detect objects within a first detection area (DA1) at a first time point (t1) in which the LiDAR device is positioned at a first detection center (DB1). In this case, the points detected by the LiDAR device at the first time point (t1) may correspond to the first detection points included in the first input frame (IFM1). Similarly, the points detected by the LiDAR device at the second time point (t2) may correspond to the second detection points included in the second input frame (IFM1).

[0110] The 'first reference frame for the third input frame' (RFM1_3) may include first motion compensation detection points (MCDP_1) corresponding to the first detection points. In this case, the first motion compensation detection points (MCDP_1) may be included in the first detection area (DA1).

[0111] The 'second reference frame for the third input frame' (RFM2_3) may include second motion compensation detection points (MCDP_2) corresponding to the second detection points. In this case, the second motion compensation detection points (MCDP_2) may be included in the second detection area (DA2).

[0112] The second overlap frame (OFM2) can be generated through a union operation on the first motion compensation detection points (MCDP_1) and the second motion compensation detection points (MCDP_2). For example, the second overlap frame (OFM2) may include overlap detection points (ODP). The overlap detection points (ODP) may be determined as the union of the first motion compensation detection points (MCDP_1) and the second motion compensation detection points (MCDP_2). That is, the overlap detection points (ODP) may include the first motion compensation detection points (MCDP_1) and the second motion compensation detection points (MCDP_2).

[0113] In one embodiment, the first motion compensation detection points (MCDP_1) and the first detection points in the first input frame (IFM1) may correspond to the same object in the motion space (OS). Similarly, the second motion compensation detection points (MCDP_2) and the second detection points in the second input frame (IFM2) may correspond to the same object in the motion space (OS).

[0114] In one embodiment, the first motion compensation detection points (MCDP_1) may include a first point (P1). The first point (P1) may be included within the first detection area (DA1) but not within the third detection area (DA3). For example, if the detection distance of the LiDAR device is set to 5 meters from the detection center, the distance between the first point (P1) and the first detection center (DB1) may be 5 meters or less, and the distance between the third detection centers (DB3) may be 5 meters or more. Accordingly, the first point (P1) may be referred to as an outbound point (OBP) for the third detection area (DA3). In this case, the overlap detection points (ODP) may not include an outbound point (OBP). However, the scope of the present disclosure is not limited thereto.

[0115] That is, according to an embodiment of the present disclosure, an overlap frame can be generated through a union operation on detection points included in different reference frames. In this case, the overlap frame may include more detection points (i.e., overlap detection points) than a single reference frame. Accordingly, when an inter-prediction operation is performed on the overlap frame and the target frame, the compression efficiency for the target frame can be improved.

[0116] FIG. 12 is a table showing a target frame and an inter-prediction frame according to an embodiment of FIG. 9. Referring to FIG. 1 to 3 and FIG. 7 to 11, the inter-prediction unit (140) can perform an inter-prediction operation based on the target frame and the inter-prediction frame.

[0117] The inter-predict unit (140) can perform an inter-predict operation based on the k-1 overlap frame (OFMk-1) when the k-th input frame (IFMk) is a target frame. For example, the inter-predict unit (140) can perform an inter-predict operation based on the first overlap frame (OFM1) when the second input frame (IFM2) is a target frame.

[0118] FIG. 13 illustrates the operation of the encoder of FIG. 3 according to another embodiment of the present disclosure. The encoding operation for the second to nth input frames (IFM2 to IFMn) will be described with reference to FIG. 1 to 3, FIG. 8, and FIG. 13. For example, FIG. 13 represents an embodiment in which the kth input frame (IFMk) (i.e., k may be an integer greater than 2) among the second to nth input frames (IFM2 to IFMn) is the target frame. The encoding operation for the first input frame (IFM1) is similar to that described above with reference to FIG. 3, so a detailed description is omitted.

[0119] Referring further to FIG. 13, the encoder (100) can receive the k-th input frame (IFMk). The motion prediction unit (110) can perform a motion prediction operation based on the k-th input frame (IFMk) and the k-1st transformed frame (CFMk-1). That is, the motion prediction unit (110) can generate the k-1st motion vector (MVk-1) based on the k-th input frame (IFMk) and the k-1st transformed frame (CFMk-1).

[0120] The motion compensation unit (120) can perform a motion compensation operation on the first to k-1 transformed frames (CFM1 to CFMk-1) based on the first to k-1 motion vectors (MV1 to MVk-1) to generate the 'first reference frame for the k-th input frame' (RFM1_k) to the 'k-1 reference frame for the k-th input frame' (RFMk-1_k). Since the operation of the motion compensation unit (120) is similar to that described above with reference to FIG. 7, a detailed description is omitted.

[0121] The union operation unit (130) can generate overlap frames by performing a union operation on any combination of 'a first reference frame for the k-th input frame' (RFM1_k) to 'a k-1 reference frame for the k-th input frame' (RFMk-1_k). For example, the union operation unit (130) can generate 'a first overlap frame for the k-th input frame' (OFM1_k) to 'a m-th overlap frame for the k-th input frame' (OFMm_k) based on any combination of 'a first reference frame for the k-th input frame' (RFM1_k) to 'a k-1 reference frame for the k-th input frame' (RFMk-1_k).

[0122] Each of the 'first overlap frame for the k-th input frame' (OFM1_k) to the 'm-th overlap frame for the k-th input frame' (OFMm_k) can be generated through a union operation of any combination of the 'first reference frame for the k-th input frame' (RFM1_k) to the 'k-1-th reference frame for the k-th input frame' (RFMk-1_k). For example, the 'first overlap frame for the k-th input frame' (OFM1_k) can be generated through a union operation of the 'first reference frame for the k-th input frame' (RFM1_k) and the 'second reference frame for the k-th input frame' (RFM2_k). The 'm-th overlap frame for the k-th input frame' (OFMm_k) may be generated through a union operation of all of the 'first reference frame for the k-th input frame' (RFM1_k) through the 'k-1-th reference frame for the k-th input frame' (RFMk-1_k). However, the scope of the present disclosure is not limited to combinations of the reference frames described above.

[0123] In one embodiment, 'the first reference frame for the k-th input frame' (RFM1_k) to 'the k-1 reference frame for the k-th input frame' (RFMk-1_k), and 'the first overlap frame for the k-th input frame' (OFM1_k) to 'the m-th overlap frame for the k-th input frame' (OFMm_k) may be included in an inter-prediction frame group. An inter-prediction frame group may refer to a set of candidate frames on which an inter-predict operation with a target frame is to be performed. An inter-prediction frame group may be stored in a memory unit (160).

[0124] The inter-predict section (140) can select one of the 'first reference frame for the k-th input frame' (RFM1_k) to the 'k-1st reference frame for the k-th input frame' (RFMk-1_k), or the 'first overlap frame for the k-th input frame' (OFM1_k) to the 'm-th overlap frame for the k-th input frame' (OFMm_k) as the inter-predict frame. That is, the inter-predict section (140) can perform an inter-predict operation on the selected frame (shown with hatching) and the k-th input frame (IFMk) (i.e., the target frame) to generate the k-th occupied code (OCk).

[0125] In one embodiment, the selected frame may be the frame most similar to the target frame among the frames in the inter-prediction frame group.

[0126] In one embodiment, the encoder (100) may measure the encoding performance for each of the frames (e.g., reference frames and overlap frames) within an inter-prediction frame group. In this case, the encoder (100) may select the frame having the highest encoding performance within the inter-prediction frame group as the inter-prediction frame. That is, according to an embodiment of the present disclosure, the frame having the highest encoding performance among the frames within the inter-prediction frame group may be selected as the inter-prediction frame.

[0127] The reconstruction unit (150) can generate the k-th transformed frame (CFMk) based on the selected frame (shown with hatching) and the k-th occupancy code (OCk). Since the k-th transformed frame (CFMk) is similar to the one described earlier with reference to FIG. 7, a detailed description is omitted.

[0128] The encoder (100) can generate a k-th bit stream (BSk) based on a k-th input frame (IFMk). In this case, the k-th bit stream (BSk) may include a k-th occupy code (OCk) and a k-1-th motion vector (MVk-1).

[0129] In one embodiment, the k-th bit stream (BSk) may further include a bit region representing information about a selected frame.

[0130] FIG. 14 shows another embodiment of step S140 of FIG. 8 according to the embodiment of FIG. 13. Referring to FIG. 1 through 3, FIG. 8, and FIG. 13 through 14, step S140 may include the following steps S241 through S247. Steps S241 and S242 are similar to steps S141 and S142 described earlier with reference to FIG. 9, respectively, so a detailed description is omitted.

[0131] In one embodiment, the reference frames generated in step S242 can be stored in an inter-prediction frame group.

[0132] In step S243, the encoder (100) can generate the 'first overlap frame for the k-th input frame' (RFM1_k) to the 'k-1st reference frame for the k-th input frame' (RFMk-1_k) by performing a union operation on any combination of these to generate the 'first overlap frame for the k-th input frame' (OFM1_k) to the 'm-th overlap frame for the k-th input frame' (OFMm_k). That is, the 'first overlap frame for the k-th input frame' (OFM1_k) to the 'm-th overlap frame for the k-th input frame' (OFMm_k) can be generated based on any different combination of the reference frames.

[0133] In one embodiment, the overlap frames generated in step S243 can be stored in an inter-prediction frame group.

[0134] In step S244, the encoder (100) can select one frame from the interprediction frame group. That is, the encoder (100) can select one of the 'first reference frame for the k-th input frame' (RFM1_k) to the 'k-1st reference frame for the k-th input frame' (RFMk-1_k), and the 'first overlap frame for the k-th input frame' (OFM1_k) to the 'm-th overlap frame for the k-th input frame' (OFMm_k).

[0135] In step S245, the encoder (100) can generate the k-th occupy code (OCk) by performing an inter-prediction operation on the selected frame and the k-th input frame (IFMk). For example, the inter-prediction unit (140) can generate the k-th occupy code (OCk) by performing an inter-prediction operation on the selected frame and the k-th input frame (IFMk).

[0136] In step S246, the encoder (100) can generate the k-th converted frame (CFMk) based on the k-th occupied code (OCk) and the selected frame.

[0137] In step S247, the encoder (100) may generate a k-th bit stream (BSk) based on a k-th occupied code (OCk) and a k-th motion vector (MVk-1). That is, the k-th bit stream (BSk) may include a k-th occupied code (OCk) and a k-th motion vector (MVk-1). Additionally, the k-th bit stream (BSk) may further include a bit region representing a frame selected in step S244, and a bit region representing reference frames used in combination with the selected frame.

[0138] FIG. 15 is a drawing showing steps S242 to S245 of FIG. 14.

[0139] Referring to FIGS. 1 to 3, FIGS. 8, and FIGS. 13 to 15, the k-th inter-prediction frame group (IPFGk) is illustrated as an inter-prediction frame group in which the k-th input frame (IFMk) is a target frame. The k-th inter-prediction frame group (IPFGk) may include a 'first reference frame for the k-th input frame' (RFM1_k) to a 'k-1 reference frame for the k-th input frame' (RFMk-1_k). The k-th inter-prediction frame group (IPFGk) may also include a 'first overlap frame for the k-th input frame' (OFM1_k) to a 'm-th overlap frame for the k-th input frame' (OFMm_k).

[0140] The inter-predict section (140) can select one of the frames in the k-th inter-predict frame group (IPFGk) to perform an inter-predict operation with the k-th input frame (IFMk). For a more concise explanation, among the frames in the k-th inter-predict frame group (IPFGk), the frame in which the inter-predict operation with the k-th input frame (IFMk) is performed will be referred to as the k-th inter-predict frame.

[0141] FIG. 16 is a table showing a target frame and an inter-prediction frame group according to an embodiment of FIG. 13. Referring to FIG. 1 to 3, FIG. 8, and FIG. 13 to 16, the inter-prediction unit (140) can perform an inter-prediction operation based on the target frame and the inter-prediction frame.

[0142] The inter-predict section (140) can perform an inter-predict operation by selecting one of the frames in the k-th inter-predict frame group (IPFGk) when the k-th input frame (IFMk) is a target frame.

[0143] FIG. 17 shows the configuration of a bit stream according to an embodiment of the present disclosure. Referring to FIGS. 1 through 17, the k-th bit stream (BSk) may include first to third bit regions (B1 to B3).

[0144] The first bit area (B1) may indicate that an encoding scheme according to an embodiment of the present disclosure has been applied. For example, the first bit area (B1) may be a flag bit indicating the encoding scheme.

[0145] The second bit region (B2) may include the k-1 motion vector (MVk-1).

[0146] The third bit area (B3) may include the k-th occupied code (OCk).

[0147] In one embodiment, the k-th bit stream (BSk) may further include a bit area representing a selected frame and a fourth bit area representing reference frames used in combination of the selected frames.

[0148] For a more concise explanation, FIG. 17 illustrates that the first to third bit regions (B1 to B3) are arranged sequentially in the k-th bit stream (BSk), but the scope of the present disclosure is not limited thereto. For example, the first to third bit regions (B1 to B3) may be arranged in a different order in the k-th bit stream (BSk), and some of the first to third bit regions (B1 to B3) may be omitted. Additionally, the first to third bit regions (B1 to B3) may be encoded through various encoding schemes (e.g., arithmetic encoding).

[0149] FIG. 18 shows a decoder according to an embodiment of the present disclosure. Referring to FIG. 18, the decoder (200) may include a motion compensation unit (220), a union operation unit (230), an inter-predict unit (240), a reconstruction unit (250), and a memory unit (260). The decoder (200) may receive first to n-th bit streams (BS1 to BSn) and generate first to n-th decoded frames (DFM1 to DFMn).

[0150] In one embodiment, the first to nth decoded frames (DFM1 to DFMn) may be identical to the first to nth converted frames (CFM1 to CFMn) described above with reference to FIGS. 7 to 12.

[0151] For a more concise explanation, an exemplary embodiment in which a decoder (200) performs a decoding operation on bit streams encoded by the encoding method described with reference to FIGS. 7 through 12 will be described. However, the scope of the present disclosure is not limited thereto, and the decoder (200) may also perform a decoding operation on bit streams encoded by the encoding method described with reference to FIGS. 13 through 16.

[0152] The functions of the motion compensation unit (220), union operation unit (230), inter-predict unit (240), reconstruction unit (250), and memory unit (260) are similar to those of the motion compensation unit (120), union operation unit (130), inter-predict unit (140), reconstruction unit (150), and memory unit (160) described earlier with reference to FIG. 3, so a detailed description is omitted.

[0153] The decoder (200) can sequentially perform decoding operations on the first to nth bit streams (BS1 to BSn). For example, the decoder (200) can perform a decoding operation on the first bit stream (BS1) to generate a first occupied code (OC1) and a first decoded frame (DFM1). Decoding operations on the second to nth bit streams (BS1 to BSn) are described in more detail with reference to FIGS. 19 to 23 below.

[0154] FIG. 19 illustrates the operation of the decoder of FIG. 18 according to an embodiment of the present disclosure. With reference to FIG. 18 and FIG. 19, decoding operations for the second to nth bit streams (BS2 to BSn) will be described. For example, FIG. 19 represents an embodiment in which a decoding operation is performed for the k-th bit stream (BSk) among the second to nth bit streams (BS2 to BSn) (i.e., k may be an integer greater than 2). Since the decoding operation for the first bit stream (BS1) is similar to that described above with reference to FIG. 18, a detailed description is omitted.

[0155] Referring further to FIG. 19, the decoder (200) can receive the k-th bit stream (BSk). The decoder (200) can generate the k-1st motion vector (MVk-1) and the k-th occupancy code (OCk) from the k-th bit stream (BSk). That is, the decoder (200) can recover the configurations illustrated in hatching in FIG. 19 from the k-th bit stream (BSk).

[0156] The decoder (200) can store the k-1st motion vector (MVk-1) in the memory unit (260). For example, when a decoding operation is performed on the k-th bit stream (BSk), the memory unit (160) can store the first motion vector (MV1) to the k-1st motion vector (MVk-1).

[0157] The motion compensation unit (220) can perform motion compensation operations for the first to k-1 decoded frames (DFM1 to DFMk-1) based on the first to k-1 motion vectors (MV1 to MVk-1). That is, the motion compensation unit (220) can generate the 'first reference frame for the k-th bit stream' (RFM1_k) to the 'k-1 reference frame for the k-th bit stream' (RFMk-1_k). Since the operation of the motion compensation unit (220) is similar to the operation of the motion compensation unit (120) described above with reference to FIG. 7, a detailed description is omitted.

[0158] In one embodiment, the 'first reference frame for the k-th bit stream' (RFM1_k) to the 'k-1st reference frame for the k-th bit stream' (RFMk-1_k) may each be identical to the 'first reference frame for the k-th input frame' (RFM1_k) to the 'k-1st reference frame for the k-th input frame' (RFMk-1_k) described above with reference to FIGS. 7 to 12.

[0159] The union operation unit (230) can generate the k-1st overlap frame (OFMk-1) by performing union operations on the 'first reference frame for the k-th bit stream' (RFM1_k) to the 'k-1st reference frame for the k-th bit stream' (RFMk-1_k). Since the operation of the union operation unit (230) is similar to the operation of the union operation unit (130) described above with reference to FIG. 7, a detailed description is omitted.

[0160] The reconstruction unit (250) can generate the k-th decoded frame (DFMk) based on the k-th overlap frame (OFMk-1) and the k-th occupancy code (OCk).

[0161] FIG. 20 shows a method of operation of the decoding device of FIG. 18 according to an embodiment of FIG. 19. Referring to FIG. 18 through FIG. 20, in step S310, the decoder (200) can generate a first decoded frame (DFM1) based on a first occupancy code (OC1). That is, the decoder (200) can generate a first decoded frame (DFM1) based on a first occupancy code (OC1) included in a first bit stream (BS1).

[0162] In step S320, the variable (k) may be set to 2. The variable (k) is used to describe that decoding operations are performed sequentially for the second to nth bit streams (BS2 to BSn) and does not limit the scope of the present disclosure.

[0163] In step S330, the decoder (200) can decode the k-th bit stream (BSk) to generate the k-th decoded frame (DFMk). For example, the decoder (200) can generate the k-th decoded frame (DFMk) based on the k-1 motion vector (MVk-1) and the k-th occupancy code (OCk) contained in the k-th bit stream (BSk). The operation of the decoder (200) in step S330 is described in more detail with reference to FIGS. 21 and FIGS. 23 below.

[0164] In step S340, it is determined whether the value of variable (k) is n. If the value of variable (k) is not n, step S350 may be performed, and variable (k) may be increased by 1. If the value of variable (k) is n, the operation of the decoder (200) may be terminated. That is, in step S340, it can be determined whether all bit streams corresponding to the same intra-period have been decoded. In step S350, the bit stream being decoded may be changed from the k-th bit stream (BSk) to the k+1-th bit stream (BSk+1).

[0165] FIG. 21 illustrates step S330 of FIG. 20 in more detail. Referring to FIG. 18 through FIG. 21, step S330 may include the following steps S331 through S333.

[0166] In step S331, the decoder (200) can generate a ‘first reference frame for the k-th bit stream’ (RFM1_k) to a ‘k-1 reference frame for the k-th bit stream’ (RFMk-1_k) based on the first to k-1 motion vectors (MV1 to MVk-1). Step S331 is similar to step S142, which was described earlier with reference to FIG. 9, so a detailed description is omitted.

[0167] In step S332, the decoder (200) may generate the k-1st overlap frame (OFMk-1) by performing a union operation on the 'first reference frame for the k-th bit stream' (RFM1_k) to the 'k-1st reference frame for the k-th bit stream' (RFMk-1_k). Step S332 is similar to step S143, which was described earlier with reference to FIG. 9, so a detailed description is omitted.

[0168] In step S333, the decoder (200) can generate the k-th decoded frame (DFMk) based on the k-th overlap frame (OFMk-1) and the k-th occupied code (OCk). Step S333 is similar to step S145, which was described earlier with reference to FIG. 9, so a detailed description is omitted.

[0169] FIG. 22 illustrates the operation of the decoder of FIG. 18 according to another embodiment of the present disclosure. With reference to FIG. 18 and FIG. 22, a decoding operation for the second to nth bit streams (BS2–BSn) encoded in the manner described with reference to FIG. 13–16 will be described. For example, FIG. 22 is representatively described for the case where a decoding operation is performed on the kth bit stream (BSk) among the second to nth bit streams (BS2–BSn) (i.e., k may be an integer greater than 2). Since the decoding operation for the first bit stream (BS1) is similar to that described above with reference to FIG. 18, a detailed description is omitted.

[0170] Referring further to FIG. 22, the decoder (200) can receive the k-th bit stream (BSk). The decoder (200) can generate the k-1st motion vector (MVk-1) and the k-th occupancy code (OCk) from the k-th bit stream (BSk). That is, the decoder (200) can recover the configurations illustrated in hatching in FIG. 22 from the k-th bit stream (BSk).

[0171] The decoder (200) can store the k-1st motion vector (MVk-1) in the memory unit (260). For example, when a decoding operation is performed on the k-th bit stream (BSk), the memory unit (160) can store the first motion vector (MV1) to the k-1st motion vector (MVk-1).

[0172] The motion compensation unit (220) can perform motion compensation operations for the first to k-1 decoded frames (DFM1 to DFMk-1) based on the first to k-1 motion vectors (MV1 to MVk-1). That is, the motion compensation unit (220) can generate the 'first reference frame for the k-th bit stream' (RFM1_k) to the 'k-1 reference frame for the k-th bit stream' (RFMk-1_k).

[0173] In one embodiment, the 'first reference frame for the k-th bit stream' (RFM1_k) to the 'k-1st reference frame for the k-th bit stream' (RFMk-1_k) may each be identical to the 'first reference frame for the k-th input frame' (RFM1_k) to the 'k-1st reference frame for the k-th input frame' (RFMk-1_k) described above with reference to FIGS. 7 to 12.

[0174] The union operation unit (230) can generate an inter-prediction frame by performing a union operation on any combination of ‘a first reference frame for the k-th bit stream’ (RFM1_k) to ‘a k-1 reference frame for the k-th bit stream’ (RFMk-1_k). For example, if the first overlap frame (OFM1_k) shown in FIGS. 13 to 16 is an inter-prediction frame for the k-th input frame (IFMk), the union operation unit (230) can generate the first overlap frame (OFM1_k) by performing a union operation on ‘a first reference frame for the k-th bit stream’ (RFM1_k) and ‘a second reference frame for the k-th bit stream’ (RFM2_k).

[0175] That is, in FIG. 22, an example is typically illustrated in which the first overlap frame (OFM1_k) is an inter-prediction frame. However, the scope of the present disclosure is not limited thereto, and any frame among the inter-prediction frame groups illustrated in FIG. 15 to FIG. 16 may be an inter-prediction frame. In this case, the union operation unit (230) may perform a union operation on any combination of 'the first reference frame for the k-th bit stream' (RFM1_k) to 'the k-1 reference frame for the k-th bit stream' (RFMk-1_k) to generate an inter-prediction frame.

[0176] In one embodiment, the k-th bit stream (BSk) may include a bit region representing information about a selected frame. In this case, the union operation unit (230) may refer to this bit region and perform a union operation on any combination of 'a first reference frame for the k-th bit stream' (RFM1_k) to 'a k-1-th reference frame for the k-th bit stream' (RFMk-1_k).

[0177] The reconstruction unit (250) can generate the k-th decoded frame (DFMk) based on the inter-prediction frame and the k-th occupied code (OCk). For example, the reconstruction unit (250) can generate the k-th decoded frame (DFMk) based on the first overlap frame (OFM1_k) and the k-th occupied code (OCk).

[0178] FIG. 23 shows another embodiment of step S330 of FIG. 20 according to the embodiment of FIG. 22. Referring to FIG. 18, FIG. 20, FIG. 22 through FIG. 23, step S330 may include the following steps S431 through S433.

[0179] In step S431, the decoder (200) can generate a ‘first reference frame for the k-th bit stream’ (RFM1_k) to a ‘k-1 reference frame for the k-th bit stream’ (RFMk-1_k) based on the first to k-1 motion vectors (MV1 to MVk-1). Step S431 is similar to step S331 described earlier with reference to FIG. 21, so a detailed description is omitted.

[0180] In step S432, the decoder (200) may generate the k-th inter-prediction frame by performing a union operation on any combination of 'the first reference frame for the k-th bit stream' (RFM1_k) to 'the k-1-th reference frame for the k-th bit stream' (RFMk-1_k). For example, the decoder (200) may generate the k-th inter-prediction frame by referring to information regarding 'reference frames used in the union operation for generating the inter-prediction frame' included in the k-th bit stream (BSk).

[0181] In step S433, the decoder (200) can generate the k-th decoded frame (DFMk) based on the k-th interprediction frame and the k-th occupied code (OCk).

[0182] FIG. 24 is a block diagram showing a lidar system according to an embodiment of the present disclosure. Referring to FIG. 24, the lidar system (1000) may include a lidar device (1100), an encoding device (1200), a storage device (1300), a decoding device (1400), and a processor (1500).

[0183] The LiDAR device (1100) may include a plurality of LiDAR sensors. The LiDAR device (1100) can collect three-dimensional point cloud data through the plurality of LiDAR sensors.

[0184] The encoding device (1200) can receive three-dimensional point cloud data from the LiDAR device (1100). The encoding device (1200) can encode the three-dimensional point cloud data and convert it into a bit stream. In one embodiment, the encoding device (1200) may be implemented in a manner similar to the encoder (100) described with reference to FIGS. 1 through 17. For example, the encoding device (1200) may convert the three-dimensional point cloud data into a plurality of frames contained within a single intra period.

[0185] The storage device (1300) can receive and store a bit stream from the encoding device (1200). For example, the storage device (1300) may include a volatile memory device such as SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), or a non-volatile memory device such as flash memory, PRAM (Phase-change Random Access Memory), MRAM (Magnetic Random Access Memory), ReRAM (Resistive Random Access Memory), FRAM (Ferroelectric Random Access Memory).

[0186] The decoding device (1400) can read a bit stream from a storage device and decode it into three-dimensional point cloud data. In one embodiment, the decoding device (1400) may be implemented in a manner similar to the decoder (200) described with reference to FIGS. 18 through 21.

[0187] The processor (1500) can receive three-dimensional point cloud data from the decoding device (1400). The processor (1500) can process operations requested by various applications using the three-dimensional point cloud data.

[0188] The above descriptions are specific embodiments for carrying out the present disclosure. The present disclosure will include not only the embodiments described above, but also embodiments that can be simply modified or easily modified. Furthermore, the present disclosure will include technologies that can be easily modified and implemented using the embodiments. Accordingly, the scope of the present disclosure should not be limited to the embodiments described above, but should be defined by the claims set forth below as well as equivalents to the claims of the present disclosure. Explanation of the symbols

[0189] ED: Encoding device PC: Point cloud data TF: Conversion unit IFM: Input Frame BS: Bitstream 100: Encoder 110: Motion prediction unit 120: Motion Compensation Unit 130: Union Operation Unit 140: Inter Predict Department 150: Reconstruction Section 160: Memory section 200: Decoder

Claims

Claim 1 An encoder that receives first to third input frames included within a first intra period and outputs a bit stream corresponding to the third input frame, wherein the encoder comprises: a motion compensation unit that generates a first reference frame corresponding to the first input frame and a second reference frame corresponding to the second input frame; a union operation unit that generates an overlap frame by performing a union operation based on the first reference frame and the second reference frame; and an inter-predict unit that generates an occupancy code by performing an inter-prediction operation on the overlap frame and the third input frame, wherein the bit stream includes the occupancy code. Claim 2 In claim 1, the first to third input frames correspond to the first to third time points of point cloud data collected by a LiDAR device outside the encoder, respectively, and the encoder. Claim 3 An encoder according to claim 2, wherein the first input frame includes first detection points for objects, the second input frame includes second detection points for objects, and the third input frame includes third detection points for objects. Claim 4 In claim 3, the encoder wherein the first reference frame includes first motion compensation detection points corresponding to the first detection points, and the second reference frame includes second motion compensation detection points corresponding to the second detection points. Claim 5 In claim 4, the overlap frame is an encoder comprising the first motion compensation detection points and the second motion compensation detection points. Claim 6 In claim 4, the first detection points are included in the first detection area of ​​the lidar device at the first time point, the second detection points are included in the second detection area of ​​the lidar device at the second time point, and the third detection points are included in the third detection area of ​​the lidar device at the third time point, and the overlap frame is an encoder comprising: motion compensation detection points included within the third detection area among the first motion compensation detection points and the second motion compensation detection points. Claim 7 In claim 6, the overlap frame is an encoder that does not include motion compensation detection points among the first motion compensation detection points and the second motion compensation detection points that are not included within the third detection area. Claim 8 In claim 1, the union operation is: an encoder performed on any combination of a plurality of reference frames including the first reference frame and the second reference frame. Claim 9 In claim 8, the bit stream comprises: an encoder including a bit region representing reference frames used in the union operation among the plurality of reference frames. Claim 10 A method of operation of an encoder that encodes first to n input frames to generate first to n bit streams, comprising the step of generating a first converted frame and a first bit stream based on the first input frame; A method of operation comprising the step of sequentially performing an encoding operation for the second to nth input frames to generate the second to nth bit streams, wherein the encoding operation for the kth input frame among the second to nth input frames comprises: a step of generating a k-1 motion vector by performing a motion prediction operation for the kth input frame and a k-1 transformed frame; a step of generating first to k-1 reference frames for the kth input frame based on the first to k-1 motion vectors; a step of generating a k-1 overlap frame by performing a union operation on the first to k-1 reference frames for the kth input frame; a step of generating a k-1 occupancy code by performing an inter-prediction operation for the k-1 overlap frame and the kth input frame; a step of generating a k-1 transformed frame based on the k-1 occupancy code and the k-1 overlap frame; and a step of generating a k-1 bit stream based on the k-1 occupancy code and the k-1 motion vector. Claim 11 In claim 10, the operation method wherein the first to nth input frames are included in the same intra-period. Claim 12 In claim 10, the operation method wherein the first to nth input frames each include detection points within different detection regions. Claim 13 In claim 10, the first to k-1 reference frames for the k-th input frame are: a method of operation in which the first to k-1 transformed frames are each the result of performing a motion compensation operation based on the first to k-1 motion vectors based on the k-th input frame. Claim 14 In claim 10, the k-1 overlap frame comprises: motion-compensated sensing points included in each of the first to k-1 reference frames for the k-th input frame. Claim 15 In claim 10, the k-th bit stream comprises a first bit area representing an encoding method, a second bit area representing the k-1 motion vector, and a third bit area representing the k-th occupancy code. Claim 16 delete Claim 17 A method of operation of a decoder for decoding first to n bit streams to generate first to n decoded frames, comprising: a step of generating a first decoded frame based on a first occupying code included in a first bit stream; and a step of sequentially performing a decoding operation for the second to n bit streams to generate the second to n decoded frames respectively, wherein the decoding operation for the k-th bit stream among the second to n bit streams comprises: a step of generating first to k-1 reference frames for the k-th bit stream based on first to k-1 motion vectors; a step of generating a k-th inter-prediction frame based on the first to k-1 reference frames for the k-th bit stream; and a step of generating a k-th decoded frame based on the k-th inter-prediction frame and the k-th occupying code, wherein the k-th inter-prediction frame is an overlap frame generated through a union operation of the first to k-1 reference frames. Claim 18 In claim 17, the k-th bit stream comprises: a first bit area representing an encoding method; a second bit area representing the k-1 motion vector; and a third bit area representing the k-th occupancy code. Claim 19 A method of operation of a decoder for decoding a first to n-th bit stream to generate a first to n-th decoded frame, comprising: a step of generating a first decoded frame based on a first occupying code included in a first bit stream; and a step of sequentially performing a decoding operation for the second to n-th bit streams to generate the second to n-th decoded frames, respectively, wherein the decoding operation for the k-th bit stream among the second to n-th bit streams comprises: a step of generating first to k-1 reference frames for the k-th bit stream based on a first to k-1 motion vector; and a step of generating a k-th interprediction frame based on the first to k-1 reference frames for the k-th bit stream. A method of operation comprising the step of generating a k-th decoded frame based on the k-th inter-prediction frame and the k-th occupying code, wherein the k-th inter-prediction frame is selected within a k-th inter-prediction frame group comprising: the first to k-1 reference frames, and a plurality of overlap frames each generated through a union operation for any combination of the first to k-1 reference frames. Claim 20 A method of operation according to claim 19, wherein the k-th bit stream comprises: a first bit area representing an encoding method; a second bit area representing the k-1 motion vector; a third bit area representing the k-th occupy code; and a fourth bit area representing reference frames among the first to k-1 reference frames used to generate the k-th interprediction frame.

Images