Encoding system, point group encoding device, encoding method, and computer program

The encoding system optimizes point cloud encoding by using motion vectors and intensity differences to selectively perform intra- or inter-frame predictions, addressing the high computational demands of conventional methods and improving processing efficiency.

JP2024165154AInactive Publication Date: 2024-11-28CANON KK
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
JP2023081067
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-16
Publication Date
2024-11-28
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Conventional encoding systems for point clouds require significant computational resources due to the use of methods like ICP for motion estimation between frames, leading to increased calculation loads.

Method used

An encoding system that integrates image and point cloud encoding, utilizing motion vectors and intensity differences to selectively perform intra- or inter-frame predictions based on predetermined thresholds, reducing the need for exhaustive calculations.

Benefits of technology

The system significantly reduces computational requirements for point cloud encoding by optimizing the selection of prediction methods, thereby enhancing processing efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024165154000001_ABST
    Figure 2024165154000001_ABST
Patent Text Reader

Abstract

To provide an encoding system or an encoding method for performing image encoding processing and point group encoding processing and allowing a reduction in the amount of operation in the point group encoding processing.SOLUTION: An encoding system has first encoding means for encoding an image, and second encoding means for encoding a point group. The second encoding means reduces the amount of operation in point group encoding processing by performing encoding processing on the basis of information used for the encoding processing in the first encoding means, for example, the size of a motion vector.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an encoding system, a point cloud encoding device, an encoding method, a computer program, and the like. [Background technology]

[0002] In recent years, sensing systems equipped with cameras and laser scanners, such as those used in self-driving vehicles, have been proposed. In such systems, a 2D image is acquired by a camera, and a 3D point cloud is measured by a laser scanner.

[0003] The information acquired by various sensors is encoded in a suitable format for data transmission. For example, encoding formats such as MPEG2 and H.264 (H.264 / MPEG-4 AVC) have been proposed for encoding two-dimensional images.

[0004] On the other hand, as a coding method for 3D data representing a three-dimensional structure such as a point cloud, the Moving Picture Experts Group (MPEG) is currently formulating G-PCC codec description v2 (Non-Patent Document 1).

[0005] For example, the following method has been proposed as an inter-frame prediction method for G-PCC. That is, when there is little change in distance between points with close laser scanning positions between frames, a method has been proposed in which the distance difference between the point with the same laser number in the previous frame and the point with the azimuth angle closest to the current azimuth angle is encoded. This method has high encoding efficiency when there is little change in distance between points in the same position between frames.

[0006] In addition, for 3D point clouds measured by installing LiDAR (Light Detection And Ranging) on ​​a moving object, a method has been proposed that uses the amount of movement of the entire point cloud between frames.

[0007] In this method, the amount of movement of the entire point cloud between frames (e.g., motion parameters expressed as rotation and translation) is estimated from the point cloud position and orientation in the previous frame and the point cloud position and orientation in the current frame.

[0008] Then, the system encodes the difference between the position of each point in the previous frame, taking into account the estimated amount of movement, and the position of each point in the point cloud in the current frame. When a LiDAR is installed on a moving body and measures the area around the moving body, the distance measured for points on a plane, such as a road area, is almost constant regardless of the amount of movement of the moving body. For this reason, this method has high encoding efficiency mainly for stationary objects other than roads. [Prior art documents] [Non-patent literature]

[0009] [Non-Patent Document 1] G-PCC codec description v2, ISO / IEC MPEG (JTC 1 / SC 29 / WG11), N18189 [Non-Patent Document 2] Lipu Zhou,Automatic Extrinsic Calibration of a Camera and a 3D LiDAR Using Line and Plane Correspondences,IEEE / RSJ IROS Summary of the Invention [Problem to be solved by the invention]

[0010] However, in the conventional technology disclosed in the above-mentioned non-patent document, one prediction mode is selected based on the results of two prediction modes, intra-frame prediction and inter-frame prediction. In addition, in order to estimate the motion between frames, it is necessary to use, for example, an ICP (Iterative Closest Point) algorithm, which increases the amount of calculation.

[0011] Therefore, one object of the present invention is to provide an encoding system capable of reducing the amount of calculation required for point group encoding processing. [Means for solving the problem]

[0012] In order to achieve the above object, the present invention provides a coding system comprising: a first encoding means for encoding an image; The present invention is characterized in that it has a second encoding means for encoding the point group, and the second encoding means performs the encoding process based on information used in the encoding process by the first encoding means. Effect of the Invention

[0013] According to the present invention, it is possible to provide an encoding system that can reduce the amount of calculation required for point group encoding processing. [Brief description of the drawings]

[0014] [Figure 1] 1 is a functional block diagram showing an example of the configuration of an encoding system according to a first embodiment of the present invention. [Diagram 2] 4 is a flowchart illustrating a prediction method determination method of a prediction method determination unit according to the first embodiment. [Diagram 3] FIG. 11 is a functional block diagram showing an example of the configuration of an encoding system according to a second embodiment of the present invention. [Figure 4] 10 is a flowchart for explaining a method for estimating a movement amount by a movement amount estimating unit according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiment. In each drawing, the same members or elements are given the same reference numerals, and duplicated descriptions are omitted or simplified.

[0016] (Embodiment 1) In the first embodiment of the present invention, an example will be described in which a camera (not shown) and a LiDAR (not shown) are fixedly installed near a building, a road, etc. In the first embodiment, it is assumed that the position and orientation between the camera and the LiDAR have been adjusted in advance using a technique such as Non-Patent Document 2, and the correspondence between pixels on the camera image and points in a three-dimensional space measured by the LiDAR has already been acquired.

[0017] Fig. 1 is a functional block diagram showing a configuration example of an encoding system according to a first embodiment of the present invention. Note that some of the functional blocks shown in Fig. 1 are realized by causing a CPU or the like as a computer (not shown) included in the encoding system to execute a computer program stored in a memory (not shown) as a storage medium.

[0018] However, a part or all of these may be realized by hardware, which may be a dedicated circuit (ASIC) or a processor (a reconfigurable processor, DSP), etc.

[0019] In addition, the functional blocks shown in Fig. 1 do not have to be built in the same housing, and may be configured by separate devices connected to each other via signal paths. The above explanation regarding Fig. 1 also applies to Fig. 3.

[0020] 1, the encoding system includes an image encoding device 100 as a first encoding means for encoding an image captured by a camera (not shown), and a point cloud encoding device 200 as a second encoding means for encoding a point cloud acquired by a LiDAR (not shown). The image encoding device 100 performs encoding in, for example, MPEG2 format.

[0021] The image coding device 100 includes a calculation unit 101, a DCT (Discrete Cosine Transform) unit 102, a quantization unit 103, a variable length coding unit 104, an inverse quantization unit 105, and an inverse DCT unit 106. The image coding device 100 also includes a calculation unit 107, a frame buffer 108, a motion compensation unit 109, and a motion vector calculation unit 110. The image coding device 100 then performs a first coding step for coding an image.

[0022] The calculation unit 101 calculates a difference value between image information captured by a camera (not shown) and predicted image information input from the motion compensation unit 109, and outputs the difference value to the DCT unit 102. The calculation unit 101 also outputs the calculated difference value between the R value of the pixel corresponding to the encoding target point and the R value of the reference pixel corresponding to the reference point to a prediction method determination unit 201 of the point group encoding device 200 described later. The DCT unit 102 performs DCT processing on the difference value input from the calculation unit 101, and outputs the resulting DCT coefficient to the quantization unit 103.

[0023] The quantization unit 103 quantizes the DCT coefficients at a predetermined quantization step, and outputs the quantized DCT coefficients to the variable-length coding unit 104 and the inverse quantization unit 105. The variable-length coding unit 104 performs variable-length coding on the quantized DCT coefficients input from the quantization unit 103 based on the motion vector detected (calculated) by the motion vector calculation unit 110, and outputs the coded data as first coded data.

[0024] The inverse quantization unit 105 inverse quantizes the quantized DCT coefficients in accordance with the quantization step in the quantization unit 103, and outputs the obtained DCT coefficients to the inverse DCT unit 106. The inverse DCT unit 106 performs inverse DCT processing on the DCT coefficients and outputs the result to the calculation unit 107.

[0025] The calculation unit 107 adds the output information from the inverse DCT unit 106 and the prediction image information from the motion compensation unit 109, and locally decodes the original image information. This locally decoded image information is output to and stored in the frame buffer 108, and is used when generating prediction image information.

[0026] The motion vector calculation unit 110 searches for a portion (hereinafter referred to as a "reference block") similar to the current block within a predetermined search range of a previously encoded frame image (hereinafter referred to as a "reference frame") stored in the frame buffer 108. Then, it detects the two-dimensional spatial movement amount from the current block to the similar portion as a motion vector. The motion vector calculation unit 110 supplies the detected motion vector to the motion compensation unit 109 and the variable length coding unit 104 .

[0027] Moreover, the motion vector calculation unit 110 outputs the detected motion vector to a prediction method determination unit 201 of the point group encoding device 200, which will be described later. Moreover, the motion vector calculation unit 110 outputs the R value of a pixel in the current block and the R value of a pixel in a reference block (hereinafter referred to as a "reference pixel"), which is a reference destination of the motion vector, to the prediction method determination unit 201. Note that the R value is a signal value corresponding to red.

[0028] The point group encoding device 200 includes a prediction method determination unit 201, a switch 202, an inter-frame prediction unit 203, a frame buffer 204, an arithmetic encoding unit 205, and an intra-frame prediction unit 206. The point group encoding device 200 performs a second encoding step of encoding a point group.

[0029] A point cloud frame acquired by a LiDAR (not shown) is input to the prediction method determination unit 201, and the prediction method determination unit 201 determines whether to perform intra-frame prediction or inter-frame prediction for each point in the point cloud frame. Here, the prediction method determination unit 201 functions as a first acquisition unit that acquires point cloud information.

[0030] The prediction method determination unit 201 makes the above-mentioned determination based on the motion vector input from the motion vector calculation unit 110, the intensity of the point to be encoded, the intensity of the point referenced by the motion vector, the R value of the pixel to be encoded, and the R value of the reference pixel.

[0031] In addition, the point to which the motion vector refers is hereinafter referred to as a "reference point". Here, the prediction method determination unit 201 functions as a second acquisition means for acquiring the first information used in the encoding by the first encoding means for encoding an image. The detailed processing procedure of the determination method will be described later.

[0032] The switch 202 functions as a selection unit for selecting whether the prediction mode to be used is intraframe prediction or interframe prediction, and selects the prediction mode determined by the prediction method determination unit 201 .

[0033] The inter-frame prediction unit 203 calculates the distance difference between the point with the same laser number in the previous frame and the point with the azimuth angle closest to the current azimuth angle, for example, in accordance with the G-PCC point cloud coding rules described in Non-Patent Document 1, and outputs the distance difference to the frame buffer 204.

[0034] The intraframe prediction unit 206 performs intraframe prediction according to, for example, the above-mentioned G-PCC point cloud coding regulation, and outputs the result to the frame buffer 204. The arithmetic coding unit 205 inputs the information output from the interframe prediction unit 203 or the intraframe prediction unit 206 via the frame buffer 204, performs arithmetic coding on the information, and outputs it as second coded data. The arithmetic coding unit 205 functions as coding means that performs coding processing on the point cloud information based on the first information.

[0035] FIG. 2 is a flowchart for explaining a method for determining a prediction method by the prediction method determination unit 201 according to the first embodiment. The processing procedure of the prediction method determination unit 201 will be described with reference to the flowchart in FIG.

[0036] The operations of the steps in the flowchart of FIG. 2 are performed in sequence by a CPU or the like serving as a computer (not shown) in the encoding system executing a computer program stored in a memory (not shown).

[0037] In step S301, the prediction method determination unit 201 calculates the magnitude of a motion vector based on the information of the motion vector input from the motion vector calculation unit 110. Next, in step S302, it is determined whether the magnitude of the motion vector is within a predetermined threshold value.

[0038] If it is determined in step S302 that the difference is within the predetermined threshold, it is determined that the movement of pixels between frames is minimal, and therefore the change in the distance between points in the point cloud corresponding to the same pixel between frames is also minimal, and the process proceeds to step S303.

[0039] On the other hand, if it is determined in step S302 that the difference is greater than the predetermined threshold, it is found that the movement of pixels between frames is large. Therefore, it is determined that the change in the distance between points in the point cloud corresponding to the same pixel between frames is also large, and the process proceeds to step S305, where the prediction mode is set to intraframe prediction, and the process flow in FIG. 2 is terminated.

[0040] In step S303, the prediction method determination unit 201 acquires information on the intensity of a reference point corresponding to the encoding target point from the frame buffer 204, and calculates the difference between the intensity of the encoding target point and the intensity of the reference point. Here, the intensity is information indicating the intensity of each point group. For example, the intensity is information indicating the intensity when the light receiving unit receives light that is emitted by the LiDAR and reflected (scattered) by an object.

[0041] If the calculated difference is within a predetermined threshold, it is determined that the encoding target point and the reference point are points indicating the same object, and the process proceeds to step 304. On the other hand, if it is determined No in step S303, the encoding target point and the reference point are similar but have different intensities. Therefore, it is determined that these points do not indicate the same object, and the process proceeds to step 305, where the prediction mode is set to intraframe prediction, and the process flow in FIG. 2 is terminated.

[0042] In step S304, the difference between the R value of the pixel corresponding to the encoding point and the R value of the reference pixel corresponding to the reference point, which is calculated by the calculation unit 101, is obtained. If the difference is within a predetermined threshold, the process proceeds to step S306, where the prediction process mode is set to inter-frame prediction, and the process flow in FIG. 2 is terminated.

[0043] On the other hand, if the result of the determination in step 304 is No, the encoding target point and the reference point have the same wavelength as Intensity, but have different R values, so it is determined that they do not represent the same object, and the process proceeds to step 305. Then, in step S305, the prediction mode is set to intra-frame prediction, and the process flow in Fig. 2 is terminated. Note that the order of steps 303 and 304 may be reversed.

[0044] In this way, the prediction method determination unit 201 obtains the motion vector output from the motion vector calculation unit 110, information on the intensities of the encoding target point and the corresponding reference point, and the R values ​​of the pixels corresponding to the encoding target point and the pixels corresponding to the reference point. Then, based on these, it is determined for each point whether the distance between the same points changes between frames.

[0045] That is, the point cloud encoding device 200, which serves as a second encoding means, selects either inter-frame prediction or intra-frame prediction based on the magnitude of the motion vector, the difference in Intensity value, and the difference in R value between the pixel of the encoding target point and the pixel of the reference point.

[0046] If it is determined that the distance indicated by corresponding points (pixels) between frames has changed, intraframe prediction is performed, and if it is determined that there has been no change, it is decided to perform interframe prediction.

[0047] Furthermore, in the first embodiment, the point cloud encoding device 200 performs the encoding process of the point cloud in the second encoding step based on the information used in the encoding process in the first encoding step in the image encoding device 100.

[0048] That is, the point cloud coding device 200 as the second coding means selects in advance either the intraframe prediction or the interframe prediction method based on information such as the magnitude of the motion vector used in the coding process in the image coding device 100. Therefore, it is possible to significantly reduce the amount of calculation compared to the conventional method in which both intraframe prediction and interframe prediction are performed and a prediction method with good coding efficiency is selected.

[0049] (Embodiment 2) Next, a second embodiment of the present invention will be described with reference to the drawings. In the second embodiment, a case where a camera and a LiDAR are installed on a moving body such as an automobile will be described. Note that the moving body is not limited to an automobile, and may be a ship, an airplane, a drone, an AGV, or the like, as long as it is movable.

[0050] Fig. 3 is a functional block diagram showing an example of the configuration of an encoding system according to embodiment 2 of the present invention. The system configuration shown in Fig. 3 includes an image encoding device 100 that encodes an image, and a point cloud encoding device 500 that encodes a point cloud, and the image encoding device 100 performs encoding in the MPEG2 format.

[0051] The configuration of the image coding device 100 is the same as that in FIG. 1, except that the output of the motion vector calculation unit 110 is supplied to a motion amount estimation unit 502, which will be described later.

[0052] The point cloud encoding device 500 includes a frame buffer 501, a motion estimation unit 502, an interframe prediction unit 503, a switch 504, an arithmetic encoding unit 505, and an intraframe prediction unit 506. A point cloud frame acquired by a LiDAR (not shown) is input to the frame buffer 501. Here, the frame buffer 501 functions as a first acquisition unit for acquiring point cloud information.

[0053] The movement amount estimation unit 502 associates the encoding target point with the reference point based on the motion vector information supplied from the motion vector calculation unit 110. Here, the movement amount estimation unit 502 functions as a second acquisition unit that acquires first information used in the encoding by a first encoding unit that encodes an image.

[0054] That is, the point cloud encoding device 500 as the second encoding means determines a reference point corresponding to the encoding target point based on the motion vector output from the image encoding device 100 as the first encoding means. Then, the movement amount estimation unit 502 estimates the movement amount of the moving object from the positional relationship (difference) between the associated points. The processing procedure of the movement amount estimation unit 502 will be described later.

[0055] The inter-frame prediction unit 503 performs inter-frame prediction, for example, as defined by G-PCC, using the amount of motion supplied from the motion estimation unit 502, and outputs the prediction result. The intra-frame prediction unit 506 inputs point information, performs intra-frame prediction, for example, as defined by G-PCC, and outputs the prediction result.

[0056] The switch 504 selects whether to input the prediction result of the intraframe prediction unit 506 to the arithmetic coding unit or the prediction result of the interframe prediction unit 503 to the arithmetic coding unit. The arithmetic coding unit 505 arithmetically codes the prediction result supplied from the switch 504 and outputs second coded data. The arithmetic coding unit 505 functions as a coding means that performs coding processing of the point group information based on the first information.

[0057] FIG. 4 is a flowchart for explaining a method for estimating a movement amount by the movement amount estimating unit 502 according to the second embodiment. The processing procedure of the movement amount estimating unit 502 will be explained with reference to the flowchart of FIG.

[0058] The operations of the steps in the flowchart of FIG. 4 are performed in sequence by a CPU or the like serving as a computer (not shown) in the encoding system executing a computer program stored in a memory (not shown).

[0059] The motion amount estimation unit 502 receives input of information about a motion vector corresponding to each point in a frame to be encoded from the motion vector calculation unit 110. In addition, the motion amount estimation unit 502 receives input of the positions of each point in the frame to be encoded and the positions of each point in the reference frame from the frame buffer 501.

[0060] In the second embodiment, in steps S601 to S604, the correspondence between the encoding target points and the reference points is obtained using points other than the road area. In steps S605 to S607, the movement amount between the points in the encoding target frame and the points in the reference frame is estimated from the correspondence between the points other than the road area. Note that the road area here is an area such as the sea surface if the moving body is a ship, and an area such as the floor surface if the moving body is an AGV, etc.

[0061] Next, the process of each of steps S601 to S604 will be described below. In step S601, the distance between the encoding target point and a point measured by the laser at the same scanning position as the encoding target point in the reference frame is calculated.

[0062] Here, the point measured by the laser at the same scanning position as the encoding target point in the reference frame means, for example, when using a rotating LiDAR, the point in the reference frame that has the same laser number as the encoding target point and the closest azimuth angle.

[0063] In step S602, it is determined whether the distance between the points is equal to or greater than a predetermined threshold value. If it is determined as Yes in step S602, it is determined that the point is not on the road area (outside the road area), and the process proceeds to step S603.

[0064] That is, since there is a large change in distance between frames of points acquired by the same laser, it is determined that the point is not an area of ​​a road, etc. If the determination is No in step S602, since there is a small change in distance between frames of points acquired by the same laser, it is determined that there is a high possibility that the point is on a road area, and the process proceeds to step S604.

[0065] In step S603, the reference point of the motion vector corresponding to the encoding target point determined to be outside the road area is set as the reference point, and the correspondence between the encoding target point and the reference point is recorded. Also, in step S604, if the processing has been completed for all pixels in the encoding target frame, the process proceeds to step S605, otherwise the process proceeds to step S601 to process the next point.

[0066] Next, in steps S605 to S607, the amount of movement between frames is estimated. The sum of distance differences between encoding target points other than road areas and their reference points (between corresponding points) is set as an objective function, and an optimization calculation is performed to estimate the amount of movement between frames that minimizes the objective function.

[0067] The process of each step from S605 to S607 will be described. In step S605, all points in the reference frame are moved by an arbitrary amount (rotation and translation). In step S606, the distance differences between the corresponding points (corresponding points) recorded in step S603 are calculated, and the sum of the distance differences is calculated.

[0068] In step S607, it is determined whether the calculated sum of distance differences is the minimum. If it is the minimum, the process proceeds to step S608, otherwise the process proceeds to step S605. In step 608, the calculated sum of distance differences is recorded as the amount of movement between frames, and the process flow in Fig. 4 is terminated. The inter-frame prediction unit 503 performs inter-frame prediction processing defined in MPEG based on the amount of movement between frames calculated in step S608.

[0069] In this manner, in this embodiment, the point cloud encoding device 500 as the second encoding means calculates the movement amount of the entire point cloud between frames from the correspondence relationship of points between frames. Also, it calculates the movement amount of the entire point cloud between frames that minimizes the distance difference between the encoding target point and the reference point.

[0070] Therefore, according to the second embodiment, the point cloud encoding device 500 can reduce the amount of calculation required when estimating the amount of movement of a point cloud between two frames.

[0071] Here, when using, for example, ICP (Iterative Closest Point) to estimate the motion between frames, it is necessary to find the nearest point for each point in the point cloud in order to find the amount of movement that most closely matches the positions of the two point clouds.

[0072] However, if the number of points in the two point groups is M and N, respectively, the amount of calculations required to find the nearest neighbors when using an algorithm such as KbTree is O(MlogN). Furthermore, repeated processing is required until the convergence condition is reached, which poses the problem of a large amount of calculations.

[0073] In contrast, in the second embodiment, when estimating the amount of movement of a point group between two frames, the correspondence between the encoding target point and the reference point is acquired in advance based on the motion vector output from the motion vector calculation unit 110 in the image encoding device 100. Therefore, when ICP is used, it is possible to reduce the amount of calculation O(MlogN) for finding the nearest point that occurs during repetitive processing, and real-time processing using ICP is also possible.

[0074] In the above embodiment, the motion between frames is estimated using ICP, but the motion between frames may be estimated using GPS (Global Positioning System).

[0075] Although the present invention has been described in detail based on the preferred embodiment, the present invention is not limited to the above embodiment, and various modifications are possible based on the spirit of the present invention, and are not excluded from the scope of the present invention. The above embodiment includes the following combinations.

[0076] (Configuration 1) An encoding system comprising a first encoding means for encoding an image and a second encoding means for encoding a point cloud, the second encoding means performing the encoding process based on information used in the encoding process by the first encoding means.

[0077] (Configuration 2) The encoding system according to configuration 1, wherein the second encoding means determines a reference point for the encoding target point based on the motion vector output from the first encoding means.

[0078] (Configuration 3) The encoding system according to configuration 1 or 2, wherein the second encoding means selects either inter-frame prediction or intra-frame prediction based on the magnitude of the motion vector output from the first encoding means.

[0079] (Configuration 4) The encoding system described in Configuration 2, characterized in that the second encoding means selects either inter-frame prediction or intra-frame prediction based on the difference in intensity values ​​between the encoding target point and the reference point.

[0080] (Configuration 5) An encoding system described in any one of configurations 2 to 4, characterized in that the second encoding means selects either inter-frame prediction or intra-frame prediction based on the difference in R value between the pixel of the encoding target point and the pixel of the reference point.

[0081] (Configuration 6) The encoding system according to any one of configurations 2 to 5, wherein the second encoding means calculates the amount of movement of the entire group of points between frames from the correspondence between points between frames.

[0082] (Configuration 7) The encoding system according to Configuration 6, wherein the second encoding means calculates the amount of movement of the entire point group between frames that minimizes the distance difference between the encoding target point and the reference point.

[0083] (Configuration 8) A point cloud encoding device comprising: a first acquisition means for acquiring point cloud information; a first encoding means for encoding an image; a second acquisition means for acquiring first information used in the encoding; and an encoding means for performing an encoding process of the point cloud information based on the first information.

[0084] (Method) An encoding method comprising a first encoding step of encoding an image and a second encoding step of encoding a point cloud, characterized in that the second encoding step performs encoding processing of the point cloud based on information used in the encoding process in the first encoding step.

[0085] (program) A computer program for controlling each means of the encoding system according to any one of configurations 1 to 7 or the point cloud encoding device according to configuration 8 by a computer.

[0086] In order to realize a part or all of the control in the above-mentioned embodiment, a computer program that realizes the functions of the above-mentioned embodiment may be supplied to an encoding device or the like via a network or various storage media. Then, a computer (or a CPU, MPU, or the like) in the encoding or the like may read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention. [Explanation of symbols]

[0087] 100: Image encoding device 110: Motion vector calculation unit 200, 500: Point cloud encoding device 201: Prediction method determination unit

Claims

1. a first encoding means for encoding an image; a second encoding means for encoding the point cloud; 2. A coding system comprising: a first coding means for coding an image based on information used in the coding process of the first coding means;

2. 2. The encoding system according to claim 1, wherein the second encoding means determines a reference point for the encoding target point based on the motion vector output from the first encoding means.

3. 2. The encoding system according to claim 1, wherein the second encoding means selects either inter-frame prediction or intra-frame prediction based on the magnitude of the motion vector output from the first encoding means.

4. 3. The encoding system according to claim 2, wherein the second encoding means selects either inter-frame prediction or intra-frame prediction based on a difference in intensity value between the encoding target point and the reference point.

5. The encoding system according to claim 2, wherein the second encoding means selects either inter-frame prediction or intra-frame prediction based on a difference in R value between the pixel of the encoding target point and the pixel of the reference point.

6. 3. The encoding system according to claim 2, wherein the second encoding means calculates an amount of movement of the entire group of points between frames from a correspondence relationship between points between frames.

7. The encoding system according to claim 6, characterized in that the second encoding means calculates the amount of movement of the entire group of points between frames that minimizes the distance difference between the encoding target point and the reference point.

8. A first acquisition means for acquiring point cloud information; A first encoding means for encoding an image includes a second acquisition means for acquiring first information used in the encoding; An encoding means for performing an encoding process of the point cloud information based on the first information; A point group encoding device comprising:

9. a first encoding step of encoding an image; a second encoding step of encoding the point cloud, An encoding method, characterized in that the second encoding step performs encoding processing of the point group based on information used in the encoding processing in the first encoding step.

10. A computer program for controlling each means of the encoding system according to any one of claims 1 to 7 or the point cloud encoding device according to claim 8 by a computer.