Analysis device, data generation method, and program

Abstract

To provide an analysis apparatus configured to generate high-quality three-dimensional data using data output form multiple sensors even if the sensors are asynchronous.SOLUTION: An analysis apparatus 19 according to the present disclosure includes: a communication unit 11 configured to receive first three-dimensional sensing data from a first sensor and second three-dimensional sensing data from a second sensor provided in a position different from a position of the first sensor; a calculation unit 12 configured to calculate a transformation parameter used to transform a reference model indicating a three-dimensional shape of a target object into a three-dimensional shape of the target object indicated by the first and second three-dimensional sensing data; a correction unit 13 configured to correct a transformation parameter in such a way that the reference model is transformed into a three-dimensional shape of the target object at a first timing based on a difference between the first timing and a second timing; and a generation unit 14 configured to generate three-dimensional data obtained by transforming the reference model using the corrected transformation parameter.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to an analysis device, a data generation method, and a program.

Background Art

[0002] In recent years, XR communication, which is a highly immersive communication that combines the real space and the virtual space, has been widely used. XR communication includes VR (Virtual Reality) communication, AR (Augmented Reality) communication, and MR (Mixed Reality) communication. In XR communication, three-dimensional data in a certain space A is transferred to a different space B, and the situation of space A is simulated and reproduced in space B. For example, the three-dimensional data acquired by a 3D (three-dimensional) camera in space A is transferred to space B, and a stereoscopic image based on the transferred three-dimensional data is displayed in space B using an AR device.

[0003] Patent Document 1 discloses a configuration of an AR system using image frames output from a color camera and a depth camera. In Patent Document 1, the color camera and the depth camera are asynchronous, and the image frames output from the color camera and the image frames output from the depth camera have different capture timings. In Patent Document 1, the image frames output from each camera are synchronized by comparing the feature points of the image frames output from the color camera and the feature points of the image frames output from the depth camera.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the AR system disclosed in Patent Document 1, there is a problem that when the object moves significantly during the timing at which each camera captures an image, the image frames output from each camera cannot be synchronized. Alternatively, even if the image frames output from each camera are synchronized when the object moves significantly during the timing at which each camera captures an image, image frames having different characteristics will be synchronized. Moving significantly may be, for example, when the amount of movement of the object exceeds a predetermined value. In this case, there is a problem that the quality of the virtual object provided by the AR system deteriorates.

[0006] One object of the present disclosure is to provide an analysis device, a data generation method, and a program capable of generating high-quality three-dimensional data using data output from a plurality of sensors even when the plurality of sensors are asynchronous.

Means for Solving the Problems

[0007] The analysis device according to the first aspect of the present disclosure includes a communication unit that receives first three-dimensional sensing data representing a result of sensing an object at a first timing from a first sensor, and second three-dimensional sensing data representing a result of sensing the object at a second timing from a second sensor installed at a position different from the first sensor, a calculation unit that calculates conversion parameters of representative points of a reference model used to convert the reference model indicating the three-dimensional shape of the object into the three-dimensional shape of the object indicated by the first three-dimensional sensing data and the second three-dimensional sensing data, a correction unit that corrects the conversion parameters so as to convert them into the three-dimensional shape of the object at the first timing based on the difference between the first timing and the second timing, and a generation unit that generates three-dimensional data obtained by converting the reference model using the corrected conversion parameters.

[0008] The data generation method according to the second aspect of the present disclosure receives first three-dimensional sensing data representing the result of sensing an object at a first timing from a first sensor, and receives second three-dimensional sensing data representing the result of sensing the object at a second timing from a second sensor installed at a position different from that of the first sensor, calculates conversion parameters of representative points of a reference model used to convert the reference model indicating the three-dimensional shape of the object into the three-dimensional shape of the object indicated by the first three-dimensional sensing data and the second three-dimensional sensing data, corrects the conversion parameters based on the difference between the first timing and the second timing so as to be converted into the three-dimensional shape of the object at the first timing, and generates three-dimensional data obtained by converting the reference model using the corrected conversion parameters.

[0009] The program according to the third aspect of the present disclosure causes a computer to receive first three-dimensional sensing data representing the result of sensing an object at a first timing from a first sensor, receive second three-dimensional sensing data representing the result of sensing the object at a second timing from a second sensor installed at a position different from that of the first sensor, calculate conversion parameters of representative points of a reference model used to convert the reference model indicating the three-dimensional shape of the object into the three-dimensional shape of the object indicated by the first three-dimensional sensing data and the second three-dimensional sensing data, correct the conversion parameters based on the difference between the first timing and the second timing so as to be converted into the three-dimensional shape of the object at the first timing, and generate three-dimensional data obtained by converting the reference model using the corrected conversion parameters.

Effect of the Invention

[0010] According to the present disclosure, it is possible to provide an analysis device, a data generation method, and a program capable of generating high-quality three-dimensional data using data output from a plurality of sensors even when the plurality of sensors are asynchronous.

Brief Description of the Drawings

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Modes for Carrying Out the Invention

[0012] (Embodiment 1) Hereinafter, embodiments of the present invention will be described with reference to the drawings. A configuration example of the analyzer 10 according to Embodiment 1 will be described with reference to FIG. 1. The analyzer 10 may be a computer device that operates by a processor executing a program stored in a memory. For example, the analyzer 10 may be a server device.

[0013] The analyzer 10 includes a communication unit 11, a calculation unit 12, a correction unit 13, and a generation unit 14. The communication unit 11, the calculation unit 12, the correction unit 13, and the generation unit 14 may be software or modules in which processing is executed by a processor executing a program stored in a memory. Alternatively, the communication unit 11, the calculation unit 12, the correction unit 13, and the generation unit 14 may be hardware such as a circuit or a chip.

[0014] The communication unit 11 receives first three-dimensional sensing data representing the result of sensing an object from a first sensor at a first timing. Further, the communication unit 11 receives second three-dimensional sensing data representing the result of sensing an object from a second sensor installed at a position different from the first sensor at a second timing.

[0015] The first sensor and the second sensor may be, for example, 3D sensors. The first sensor is installed at a location different from the second sensor, and each sensor senses the object from a different angle. Sensing may be, for example, imaging an object using a camera. Sensing may be rephrased as capturing. The first sensor and the second sensor are not synchronized in the timing for sensing the object. Timing may be rephrased as time or hour.

[0016] The first sensor and the second sensor generate 3D sensing data by sensing an object. The 3D sensing data is data that includes depth data in addition to 2D image data. The depth data is data indicating the distance from the sensor to the object. The 2D image data may be, for example, RGB (Red Green Blue) image data.

[0017] The calculation unit 12 calculates conversion parameters of representative points of a reference model used to convert a reference model indicating the 3D shape of an object into the 3D shape of the object indicated by the first 3D sensing data and the second 3D sensing data. The reference model indicating the 3D shape may be, for example, mesh data, a mesh model, or polygon data, and may be referred to as 3D data. The reference model may be generated, for example, using 3D sensing data generated by the first sensor and the second sensor sensing the object. The reference model may be generated for each 3D sensing data generated by each of the first sensor and the second sensor. Or, the reference model may be generated using data obtained by integrating or synthesizing the 3D sensing data generated by the first sensor and the second sensor.

[0018] The conversion parameters may be, for example, parameters for converting a point cloud of the reference model into a point cloud of the 3D shape of the object indicated by the first 3D sensing data and the second 3D sensing data.

[0019] The correction unit 13 corrects the conversion parameters so as to convert them into the 3D shape of the object at the first timing based on the difference between the first timing and the second timing. The difference between the first timing and the second timing may also be referred to as a time difference or a time lag.

[0020] For example, assume that the time indicated by the second timing is later than the time indicated by the first timing. In this case, it is assumed that the amount of change when converting the reference model to the three-dimensional shape of the object at the second timing is larger than the amount of change when converting the reference model to the three-dimensional shape of the object at the first timing. Therefore, the conversion parameter may be corrected using the difference between the first timing and the second timing so as to be the amount of change when converting the reference model to the three-dimensional shape of the object at the first timing.

[0021] The generation unit 14 generates three-dimensional data obtained by converting the reference model using the corrected conversion parameter. The reference model converted using the corrected conversion parameter shows the three-dimensional shape of the object at the first timing.

[0022] As described above, the analysis device 10 according to the first embodiment generates three-dimensional data showing the three-dimensional shape of the object by applying the conversion parameter to the reference model. Further, the analysis device 10 converts the conversion parameter based on the timing difference in sensing between the first sensor and the second sensor. That is, the conversion parameter is corrected so that the amount of change when the conversion parameter is applied to the reference model indicates the amount of change at the first timing. Thereby, the analysis device 10 can convert the reference model into three-dimensional data showing the three-dimensional shape shown by the three-dimensional sensing data sensed at different timings.

[0023] (Second Embodiment) Subsequently, a configuration example of the AR communication system according to the second embodiment will be described with reference to FIG. 2. The AR communication system in FIG. 2 includes an analysis device 20, cameras 30 to 33, an access point device 40, and a user terminal 50. The analysis device 20 corresponds to the analysis device 10 in FIG. 1. In FIG. 2, a configuration example of an AR communication system including four cameras is shown, but the number of cameras is not limited to four.

[0024] Cameras 30 to 33 are specific examples of 3D sensors, and other devices that acquire three-dimensional data of an object may be used instead of the cameras. Cameras 30 to 33 are, for example, 3D cameras, and generate point cloud data of the object. The point cloud data has values related to the positions on the two-dimensional plane in the images generated by the respective cameras and the distances from the cameras to the object. The image indicated by the point cloud data is an image showing the distance from the camera to the object, and may also be referred to as a depth image or a depth map, etc. Cameras 30 to 33 photograph the object and transmit the point cloud data, which is the photographed data, to the analysis device 20 via a network.

[0025] The analysis device 20 receives the point cloud data from cameras 30 to 33 and generates analysis data necessary to display the object on the user terminal 50. The analysis data may be, for example, three-dimensional data indicating the object.

[0026] The access point device 40 is, for example, a communication device that supports wireless LAN (Local Area Network) communication, and may also be referred to as a wireless LAN master unit. In contrast, the user terminal 50 that performs wireless LAN communication with the access point device 40 may also be referred to as a wireless LAN slave unit.

[0027] The user terminal 50 may be, for example, an xR device, and specifically, may be an AR device. The user terminal 50 performs wireless LAN communication with the access point device 40 and receives the analysis data generated in the analysis device 20 via the access point device 40. Note that the user terminal 50 may receive the analysis data from the analysis device 20 without performing wireless LAN communication, via a mobile network, or may receive the analysis data from the analysis device 20 via a fixed communication network such as an optical network.

[0028] Next, a configuration example of the analyzer 20 will be described with reference to FIG. 3. The analyzer 20 includes a parameter calculation unit 21, a 3D model update unit 22, a communication unit 23, a parameter correction unit 24, and a 3D data generation unit 25. The parameter calculation unit 21, the 3D model update unit 22, the communication unit 23, the parameter correction unit 24, and the 3D data generation unit 25 may be software or modules whose processing is executed by executing a program. Alternatively, the parameter calculation unit 21, the 3D model update unit 22, the communication unit 23, the parameter correction unit 24, and the 3D data generation unit 25 may be hardware such as a circuit or a chip. The parameter calculation unit 21 corresponds to the calculation unit 11 in FIG. 1. The communication unit 23 corresponds to the communication unit 13 in FIG. 1. The parameter correction unit 24 corresponds to the correction unit 13 in FIG. 1. The 3D data generation unit 25 corresponds to the generation unit 14 in FIG. 1.

[0029] The communication unit 23 receives point cloud data including the object from the cameras 30 to 33. The communication unit 23 outputs the received point cloud data to the 3D model update unit 22. If the 3D model update unit 22 has not created a reference model for the object so far, it creates a reference model. For example, the 3D model update unit 22 may synthesize the point cloud data received from each camera to generate point cloud data including the object. Synthesizing the point cloud data may also be referred to as integrating the point cloud data. For example, for the integration of the point cloud data, ICP (Iterative Closest Point) or pose estimation processing may be used. For the pose estimation processing, for example, a PnP (Perspective-n-Point) solver may be used. Further, the 3D model update unit 22 generates a reference model of the object from the point cloud data. The reference model is composed of a point cloud and representative points having conversion parameters of a part thereof. By transforming the reference model, it becomes possible to obtain a new point cloud, and it also becomes possible to generate mesh data from the obtained point cloud. Generating the mesh data may also be referred to as constructing a mesh. The mesh data is data that indicates the three-dimensional shape of the surface of the object by triangular faces or quadrilateral faces formed by combining each vertex with the points included in the point cloud data. Alternatively, the 3D model update unit 22 may generate a reference model of the object for each point cloud data without integrating the point cloud data received from each camera.

[0030] The 3D model update unit 22 stores the reference model in a memory or the like within the analyzer 20. Further, the 3D model update unit 22 transmits data regarding the reference model to the user terminal 50 via the communication unit 23 and the access point device 40. The data regarding the reference model may include data indicating vertices having vertex positions and transformation parameters. Alternatively, if the user terminal 50 can generate mesh data from the point cloud data in the same manner as the 3D model update unit 22, the 3D model update unit 22 may transmit to the user terminal 50 the point cloud data obtained by synthesizing the point cloud data generated by each camera. When the user terminal 50 receives the reference model, or when the user terminal 50 generates the reference model from the point cloud data in the same manner as the 3D model update unit 22, the analyzer 20 and the user terminal 50 share the same reference model.

[0031] After the reference model is generated in the 3D model update unit 22, when the communication unit 23 receives point cloud data from the cameras 30 to 33, the parameter calculation unit 21 calculates transformation parameters using the reference model and the point cloud data. Here, the parameter calculation unit 21 may generate, in the same manner as the 3D model update unit 22, point cloud data obtained by synthesizing the point cloud data generated by each camera. Alternatively, when the point cloud data is synthesized in the 3D model update unit 22, the parameter calculation unit 21 may receive the synthesized point cloud data from the 3D model update unit 22.

[0032] Further, the parameter calculation unit 21 may calculate transformation parameters using each point cloud data without synthesizing the point cloud data generated by each camera.

[0033] The parameter calculation unit 21 extracts representative points from the vertices of the reference model and generates representative point data indicating the representative points. The parameter calculation unit 21 may randomly extract representative points from all the vertices included in the reference model, or may execute the k-means method to extract representative points so that the representative points are evenly arranged within the object.

[0034] The parameter calculation unit 21 calculates transformation parameters so that the vertices of the reference model represent the three-dimensional shape of the object indicated by the point cloud data received from the cameras 30 to 33 after the reference model is generated. The three-dimensional shape of the object indicated by the point cloud data received from the cameras 30 to 33 may be the three-dimensional shape of the object indicated by the point cloud data after synthesizing the point cloud data of each camera. Alternatively, the three-dimensional shape of the object indicated by the point cloud data received from the cameras 30 to 33 may be the three-dimensional shape of the object indicated by each point cloud data without synthesizing the point cloud data of each camera.

[0035] The transformation parameters are calculated for each representative point. When there are n (n is an integer of 1 or more) representative points, the transformation parameters are the transformation parameter W k as shown. k is a value for identifying the representative point and can take values from 1 to n. All transformation parameters may be shown as transformation parameter W = [W1, W2, W3, ··· W n . The transformation parameter W k includes the rotation matrix R k and the translation matrix T k , and W k = [R k |T k may be shown. Let the vertex of the reference model be vertex v i . Then, the transformed vertex v' i is v' i = Σα k (v i )·Q k (v i ), Q k (v i ) = R k ·(v i ― v k ) + v k + T k may be shown. "·" represents multiplication. Here, v k refers to the representative point of the reference model, and α k is a function for calculating the proximity between the representative point v k and the vertex v i , and the sum is 1, that is, Σα k (v i ) = 1. α k (v i) is v i The closer it is to the representative point, the larger the value becomes. In other words, the conversion closer to the representative point will occur. This Q k (v i ) is an example of applying rotation and translation to a point, and other expressions for applying rotation and translation may also be used.

[0036] Here, an example of calculating the conversion parameter W = [W1, W2, W3, ··· W n in the parameter calculation unit 21 will be described. The left diagram in FIG. 4 shows the projection of the vertex v' i after conversion onto the coordinates (Cx, Cy) on the two-dimensional image. The two-dimensional image may be an image on the X-Y plane of the image shown using the point cloud data. Also, the right diagram in FIG. 4 shows the point H i in the three-dimensional space corresponding to the coordinates (Cx, Cy) on the two-dimensional image of the point cloud data received after generating the reference model. For the projection of the vertex v' i after conversion onto the coordinates (Cx, Cy) on the two-dimensional image, the internal parameters of the camera are used and it is performed to be the same as the coordinate system on the two-dimensional image of the camera. Regarding the distance data of the vertex v' i after conversion, let it be v' i (D), and regarding the distance data of the point H k , let it be H k (D). The parameter calculation unit 21 calculates the conversion parameter W = [W1, W2, W3, ··· W n such that Σ|v' i (D) - H i (D)| 2 is minimized. |v' such that Σ|v' i (D) - H i (D)| is the absolute value of v' i (D) - H i (D). Here, in Σ|v' i (D) - H i (D)| 2 , i can take values from 1 to m. m represents the number of vertices of the reference model.

[0037] The parameter calculation unit 21 outputs the conversion parameter W and the point cloud data generated in each camera received from the communication unit 23 to the parameter correction unit 24. The parameter correction unit 24 corrects the conversion parameter W, and the 3D data generation unit 25 converts the reference model using the corrected conversion parameter. The 3D data generation unit 25 generates the mesh data of the object using the point cloud data after converting the reference model.

[0038] The 3D data generation unit 25 applies the conversion parameter W to each representative point included in the reference model to calculate the converted representative point. Here, the 3D data generation unit 25 converts the representative point using the conversion parameter W, and for vertices other than the representative point, it converts them using the conversion parameter W used for the neighboring representative points. Thus, converting the position of the representative point using the conversion parameter and converting the positions of vertices other than the representative point using the conversion parameter used at the neighboring representative points may be referred to as non-rigid transformation.

[0039] Here, the non-rigid transformation will be described with reference to FIG. 5. FIG. 5 shows the vertices included in the reference model being transformed and the positions of the vertices in the 3D data regarding the three-dimensional shape indicated by the point cloud data received after the reference model is generated. The double circles in FIG. 5 indicate the representative points, and the single circles indicate the vertices other than the representative points. For the sake of simplicity of explanation, the case where only these three points exist in the reference model will be described. Here, it is assumed that the distances between the vertex v other than the representative point and the two representative points are the same. At this time, α k (v) = 0.5 (k = 1, 2), and after conversion, v' is v'=(R1·(v - v1)+v1+T1+R2·(v - v2)+v2+T2) / 2. " / " represents division. For the representative point v k , so as not to receive neighboring representative points, α k’ (v k ) = 0.0 (k≠k'), α k’ (v k) = 1.0 (k == k’) may be set, or the conversion formula may be used as with other vertices.

[0040] Returning to FIG. 3, the 3D data generation unit 25 performs non-rigid transformation to transform the point cloud of the reference model, and generates mesh data using the transformed point cloud.

[0041] Next, the update process of the reference model executed in the 3D model update unit 22 will be described. The 3D model update unit 22 applies the inverse transformation parameter W of the transformation parameter W calculated by the parameter calculation unit 21 to the point cloud data received after the reference model is generated. -1 The inverse transformation parameter W -1 The point cloud data to which the inverse transformation parameter W is applied is transformed into point cloud data showing substantially the same three-dimensional shape as the reference model. The reference model and the inversely transformed point cloud are updated with a newly generated point cloud obtained by performing a process such as averaging as a new reference model. Averaging may be performed, for example, by representing the point cloud in a data structure based on TSDF (Truncated Signed Distance Function) and then performing weighted averaging considering time. The representative points may be newly generated based on the updated reference model, or only the locations with significant changes may be updated.

[0042] Here, with reference to FIG. 6, the difference between the point cloud data after the inverse transformation parameter W -1 is applied and the vertices of the reference model will be described. The left diagram in FIG. 6 shows the mesh data included in the reference model before update. The double circles indicate the representative points. The mesh data is shown as triangular faces using three vertices. The right diagram shows the inverse transformation parameter W -1Mesh data using the vertices after [a certain condition] is applied is shown. The mesh data in the right figure has one more vertex added compared to the left figure. In Figure 6, the added vertex is shown as a representative point, but it doesn't have to be a representative point. By updating the reference model using the newly added vertex, the accuracy of the reference model can be improved. Each time the update is repeated, the information indicating the shape of the reference model increases, and the accuracy of representing the object improves.

[0043] The 3D model update unit 22 transmits the difference data between the reference model before update and the reference model after update to the user terminal 50 via the communication unit 23. The difference data may include vertices newly added or deleted in the reference model after update, data regarding representative points, and further data regarding the vertices of newly added triangular or quadrilateral faces.

[0044] Next, with reference to Figure 7, the correction process of the conversion parameters in the parameter correction unit 24 will be described. The horizontal arrows in Figure 7 indicate the passage of time. The further to the right the arrow goes, the more time has passed. Camera #1 and Camera #2 represent, for example, any one of cameras 30 to 33. In Figure 7, for ease of explanation, the correction process of the conversion parameters when using two cameras is shown.

[0045] D1(1) shown in the time series of Camera #1 represents the 3D sensing data generated by Camera #1 at timing 1. The "1" at timing 1 indicates the identification name of the timing, not a time such as 1 second. The same applies to the description of the timing described below. The 3D sensing data may be, for example, point cloud data. D1(2) represents the 3D sensing data generated by Camera #1 at timing 2.

[0046] D2(1 + a) shown in the time series of Camera #2 represents the 3D sensing data generated by Camera #2 at timing 1 + a. D2(2 + a) represents the 3D sensing data generated by Camera #2 at timing 2 + a. a is a value greater than 0, for example, and 1 + a may indicate a seconds after timing 1.

[0047] Reference Model #1 is generated based on the 3D sensing data generated by Camera #1. Specifically, Reference Model #1 may be mesh data of a 3D shape by combining each point of the 3D sensing data which is point cloud data. For example, Reference Model #1 is generated using D1(1).

[0048] Reference Model #2 is generated based on the 3D sensing data generated by Camera #2. For example, Reference Model #2 is generated using D2(1 + a). Since Camera #1 and Camera #2 are installed at different positions, the display content of the object photographed by each camera is also different. For example, D1(1) may show a front image of the object, and D2(1 + a) may show a back image of the object. In this case, Reference Model #1 may be a model showing the front of the object, and Reference Model #2 may be a model showing the back of the object.

[0049] Point Cloud Data #1 is D1(2) generated by Camera #1, and W1(2) represents the transformation parameters to be applied to Reference Model #1. Specifically, W1(2) is the transformation parameter for transforming Reference Model #1 into the 3D shape of the object indicated by D1(2). In other words, W1(2) causes Reference Model #1 to transition (transform) into the 3D shape of the object indicated by D1(2). The Reference Model #1 transformed using W1(2) may be shown as point cloud data, for example, or may be shown as 3D data which is mesh data.

[0050] The point cloud data #2 is D2(2 + a) generated by camera #2, and W2(2 + a) indicates the conversion parameters applied to the reference model #2. Specifically, W2(2 + a) is the conversion parameter for converting the reference model #2 into the three-dimensional shape of the object indicated by D2(2 + a).

[0051] Here, let the time difference between the timing when D1(2) was created and the timing when D2(2 + a) was created be a seconds. That is, the three-dimensional shape indicated by the three-dimensional data converted from the reference model #2 using W2(2 + a) shows the shape after a seconds have elapsed from the three-dimensional shape indicated by the three-dimensional data converted from the reference model #1 using W1(2).

[0052] The parameter correction unit 24 corrects the conversion parameter W2(2 + a) to W2(2) so that the three-dimensional data converted from the reference model #2 using W2(2 + a) shows a three-dimensional shape at substantially the same timing as the timing when D1(2) was generated. Specifically, the parameter correction unit 24 calculates the conversion parameter W2(2) at timing 2 using b, which is the time difference between D2(1 + a) and D2(2 + a), and a. Specifically, for the rotation matrix and translation matrix of the representative point of the conversion parameter W2(2 + a), let them be R k , T k . For R k , when using roll φ k , pitch θ k , and yaw ψ k as parameters, and R k = f(φ k , θ k , ψ k ), the rotation matrix of W2(2) is f(φ k ·(b - a) / b, θ k ·(b - a) / b, ψ k ·(b - a) / b). Also, the translation matrix is T k· It becomes (b - a) / b. The reference model after conversion shows the three-dimensional shape of the object indicated by the point cloud data sensed by camera #2 at a timing substantially the same as the timing when D1(2) was generated. In other words, it can be said that W2(2) is a conversion parameter that rewinds the converted point cloud data obtained using W2(2 + a) by a seconds. In FIG. 7, the reference model #2 after conversion using W2(2) is shown as the adjusted point cloud data.

[0053] The 3D data generation unit 25 integrates the point cloud data #1 converted from the reference model #1 using the conversion parameter W1(2) and the adjusted point cloud data converted from the reference model #2 using the conversion parameter W2(2). For example, the 3D data generation unit 25 integrates the point cloud data #1 and the adjusted point cloud data using ICP or the pose information of each camera. Further, the 3D data generation unit 25 generates mesh data using the integrated point cloud data. The mesh data generated in this way shows the three-dimensional shape of the object at the timing when D1(2) was generated.

[0054] Here, the reference model #1 is updated using the point cloud data obtained by applying the inverse conversion parameter W -1 1(2) to the point cloud data #1. Further, the reference model #2 is updated using the point cloud data obtained by applying the inverse conversion parameter W -1 2(2 + a) to the point cloud data #2.

[0055] Subsequently, with reference to FIG. 8, the correction process of the conversion parameter in the parameter correction unit 24 different from that in FIG. 7 will be described. The reference model #1 is generated using the three-dimensional sensing data D1(1) generated by camera #1 in the same manner as in FIG. 7. The three-dimensional sensing data may be, for example, point cloud data. The reference model #2 is also generated using the three-dimensional sensing data D2(1 + a) generated by camera #2 in the same manner as in FIG. 7.

[0056] The conversion parameters W1(2) and W2(2+a) are the same as those in FIG. 7, and the reference models #1 and #2 are converted into the three-dimensional shapes indicated by D1(2) or D2(2+a), respectively.

[0057] The parameter correction unit 24 corrects the conversion parameter W2(2+a) so that it represents the three-dimensional shape indicated by the three-dimensional sensing data captured by camera #2 at a timing substantially the same as the timing when D1(1) was generated using W2(2+a). The corrected conversion parameter is calculated by multiplying the inverse conversion parameter W ―1 2(2+a) by the coefficient M.

[0058] The time difference between D1(1) and D2(1+a) is a. Further, let the time difference between D2(1+a) and D2(2+a) be b. W ―1 For the rotation matrix and translation matrix of the representative point k in W k 2(2+a), let them be R k and T k respectively. For R k with roll φ k , pitch θ k , and yaw ψ k as parameters, when R k =f(φ k , θ k , ψ ―1 ), the rotation matrix and translation matrix of W k 2(2+a)' that is rewound at a timing substantially the same as the timing when D1(1) was generated are R k ' and T k ' respectively. In this case, R k ' =f(φ k ·(b+a) / b, θ k ·(b+a) / b, ψ k ·(b+a) / b), and the translation matrix is T ―1 ·(b+a) / b. The 3D model update unit 22 generates the reference model #2 of the object captured by camera #2 at a timing substantially the same as the timing when D1(1) was generated, using the point cloud data after being converted using W

[0059] Here, the generation of the integrated reference model will be described with reference to FIG. 9. Similar to FIGS. 7 and 8, the horizontal arrows in FIG. 9 indicate the passage of time. The further the arrow progresses to the right, the more time has elapsed.

[0060] The solid line on the horizontal arrow of the reference model #1 in FIG. 9 indicates the reference model #1 of the object at the timing when D1(1) was generated in camera #1. Also, the solid line on the horizontal arrow of the reference model #2 indicates the reference model #2 of the object at the timing when D2(1 + a) was generated in camera #2. The dotted line on the horizontal arrow of the reference model #2 indicates the reference model #2 of the object indicated by the point cloud data generated in camera #2 at a timing substantially the same as the timing when D1(1) was generated in camera #1.

[0061] The 3D model update unit 22 integrates the reference model #1 of the object at the timing when D1(1) was generated in camera #1 and the reference model #2 of the object regarding camera #2 at the timing when D1(1) was generated in camera #1. The 3D model update unit 22 integrates the reference model #1 and the reference model #2 to generate an integrated reference model. The 3D model update unit 22 generates an integrated reference model by integrating the two reference models using, for example, ICP or the pose information of the camera. Also, the integration of the reference models may be performed on the data structure based on TSDF (Truncated Signed Distance Function). The integrated reference model is a reference model generated using the integrated point cloud data obtained by integrating the point cloud data captured in camera #1 and camera #2 at the timing when D1(1) was generated in camera #1.

[0062] Next, the process of calculating the conversion parameter W(t) with reference to the integrated reference model using FIG. 9 will be described. W(t) is the conversion parameter at timing t, and W(t)=[W1, W2, W3, ··· W nIt is a set of conversion parameters for each representative point. D1(t) is the point cloud data generated at timing t in camera #1, and D2(t+a) is the point cloud data generated at timing t+a in camera #2. Also, let the time difference between D1(1) and D1(t) be t.

[0063] The parameter correction unit 24 uses the converted vertex v' of the reference model with the conversion parameter W(t) i (t) as v' i (t)=Σα k (v i )·Q k (v i )、Q k (v i )=R k (t, g(t))·(v i - v k )+v k +T k (t, g(t)) 、 R k (t, g(t))-f(g(t)×φ k (t), g(t)×θ k (t), g(t)×ψ k (t)), T k (t, g(t))=g(t)×T k、 is calculated using. v' i (t) indicates the vertex after conversion of the integrated reference model. Further, the parameter correction unit 24 projects the converted vertex v' i (t) onto the coordinates (Cx, Cy) on the 2D image in the same way as the method of projecting D1(t) into the 2D space. When D1(t) is composed of a 2D depth image, the converted vertex v' i (t) is projected so as to be in the same coordinate system as the depth image. Let the point in the 3D space corresponding to (Cx, Cy) in D1(t) be H t and. When projecting onto the same 2D image as the 2D image that D1(t) has, the parameter correction unit 24 sets g(t)=1. Let the conversion parameter at this time be W(t). Also, the parameter correction unit 24 projects the converted representative point v' i(t + a) is projected onto the coordinates (Cx, Cy) on the two-dimensional image in the same way as the method of projecting D2(t + a) into the two-dimensional space. Here, when projecting, the parameter correction unit 24 sets g(t) = (t + a) / t, and v’ k calculates (t + a). Let the conversion parameter at this time be W(t + a).

[0064] When the integrated reference model is converted using the conversion parameter W(t), it is converted to a point indicating the three-dimensional shape of the object at a timing substantially the same as the timing when D1(t) was generated. However, the point cloud data #2 is D2(t + a) generated, for example, a seconds after D1(t) was generated. When the parameter correction unit 24 projects onto the same two-dimensional image as the two-dimensional image that D2(t + a) has, by setting g2 = (t + a) / t, the reference model can be converted to a point indicating the three-dimensional shape of the object at the timing when D2(t + a) was generated.

[0065] Let the point in the three-dimensional space corresponding to (Cx’, Cy’) in D2(t + a) be H’ t The parameter correction unit 24 calculates Σ{(v’ k (t, D) - H k (t, D)) 2 + (v’ k (t + a, D) - H’ k (t + a, D)) 2} to be minimized such that the conversion parameter W(t) = [W1, W2, W3, ··· W n is calculated. k can take values from 1 to n. The (t, D) in v’ k (t, D) and H k (t, D) indicates that it is the distance data from the two-dimensional image plane at the timing t.

[0066] The 3D data generation unit 25 generates three-dimensional data of the object at the timing t by converting the integrated reference model using the conversion parameter W(t).

[0067] Here, the integrated reference model uses the point cloud data obtained by applying the inverse transformation parameter W -1 (t) to the point cloud data #1 (D1(t)), and the point cloud data obtained by applying the inverse transformation parameter W -1 (t + a) to the point cloud data #2 (D2(t + a)) for updating.

[0068] Subsequently, the process of calculating the transformation parameter W(t + 1) based on the integrated reference model using FIG. 10 will be described. In FIG. 10, the process of calculating W(t + 1) using W(t) calculated in FIG. 9 will be described. For the transformation parameter for D1(t + 1), W(t + 1) is calculated in the same manner as W(t) calculated in FIG. 9. Also, for W(t + 1 + a), R k (t + 1 + a, g(t + 1 + a)), T k (t + 1 + a, g(t + 1 + a)) are used, and R k (t + 1 + a, g(t + 1 + a)) - f(g(t + 1 + a)×(φ k (t + 1) - φ k (t)) + φ k (t), g(t + 1 + a)×(θ k (t + 1) - θ k (t)) + θ k (t + 1), g(t + 1 + a)×(ψ k (t + 1) - ψ k (t)) + ψ k (t + 1)), T k (t + 1 + a, g(t + 1 + a)) = g(t + 1 + a)×(T k (t + 1, g(t + 1)) - T k (t, g(t))) + T k (t, g(t)) are used for calculation. Here, g(t + 1 + a) = a / b.

[0069] The parameter correction unit 24 uses the vertex v' i (t + 1) after transformation using the transformation parameter W(t + 1), where v' i (t + 1) = Σα k (v i )·Q k (v i ), Q k (vi ) = R k ·(v i - v k ) + v k + T k is calculated using. v k (t) represents the representative points included in the point cloud of the integrated reference model. Further, the parameter correction unit 24 projects the vertex v' i (t + 1) onto the coordinates (Cx, Cy) on the same two-dimensional image as the two-dimensional image that D1(t + 1) has. Further, the point in the three-dimensional space corresponding to (Cx, Cy) within D1(t + 1) is H i Let it be. Also, the parameter correction unit 24 projects the vertex v' i (t + 1) onto the coordinates (Cx', Cy') on the same two-dimensional image as the two-dimensional image that D2(t + 1 + a) has. Here, when the parameter correction unit 24 projects onto the same two-dimensional image as the two-dimensional image that D2(t + 1 + a) has, it calculates v' i (t + 1 + a) using W(t + 1 + a). On the other hand, when the parameter correction unit 24 projects onto the same two-dimensional image as the two-dimensional image that D1(b + 1) has, g = g1 = 1 is set.

[0070] Let the point in the three-dimensional space corresponding to (Cx', Cy') in D2(t + 1 + a) be H' i Let it be. The parameter correction unit 24 calculates the transformation parameter W(t + 1) = [W1, W2, W3, ··· W i (t + 1, D) - H i (t + 1, D)) 2 +(v' i (t + 1 + a, D) - H' i (t + 1 + a, D)) 2} such that it becomes minimum. n is calculated.

[0071] Next, with reference to FIG. 11, the process flow regarding the generation of mesh data and the update of the reference model in the configuration described in FIG. 7 will be described. First, the communication unit 23 receives point cloud data including the object from the cameras 30 to 33 (S11). Next, the 3D model update unit 22 generates a reference model of the object photographed for each camera using the acquired point cloud data (S12). For example, the 3D model update unit 22 generates mesh data indicating the three-dimensional shape of the object using triangular or quadrilateral faces formed by combining the respective points included in the point cloud data.

[0072] Next, after the communication unit 23 acquires the point cloud data from each camera in step S11, the communication unit 23 further acquires the point cloud data from each camera (S13). The newly received point cloud data is data indicating the substantially real-time three-dimensional shape of the object and may be referred to as real-time data. Next, the parameter calculation unit 21 calculates conversion parameters for converting the reference model generated by the 3D model update unit 22 into the three-dimensional shape of the object indicated by the point cloud data acquired in step S13 (S14). The conversion parameters are calculated for each representative point among the vertices constituting the reference model.

[0073] Next, the parameter correction unit 24 corrects the conversion parameters calculated in step S14 (S15). For example, the parameter correction unit 24 corrects the conversion parameters related to camera #2 included in the cameras 30 to 33 so as to convert them into the three-dimensional shape of the object at substantially the same timing as the timing when the point cloud data was generated in camera #1. Correcting the conversion parameters may mean decreasing or increasing the amount of change in the reference model related to each camera. For example, there may be a case where the timing at which camera #2 generates the point cloud data is later than the timing at which camera #1 generates the point cloud data. In this case, the conversion parameters are corrected so as to decrease the amount of change in the reference model related to camera #2 so as to match the timing at which camera #1 generates the point cloud data.

[0074] Next, the 3D data generation unit 25 integrates the point cloud data after converting the vertices that constitute the reference model for each camera, and generates mesh data (S16). The point cloud data after conversion for each camera shows the three-dimensional shape of the object at substantially the same timing.

[0075] Next, the 3D model update unit 22 updates the reference model using the calculated conversion parameters (S17). For example, the 3D model update unit 22 updates the reference model using the point cloud data obtained by applying the inverse conversion parameters to the point cloud data generated by each camera.

[0076] Subsequently, with reference to FIG. 12, the flow of processing related to the generation of mesh data and the update of the reference model in the configuration described in FIGS. 8 and 9 will be described. Steps S21 to S24 are the same as steps S11 to S14 in FIG. 11, and thus detailed description thereof will be omitted.

[0077] The 3D model update unit 22 generates an integrated reference model in step S25 (S25). Specifically, there is a difference in the timing of generating the point cloud data between camera #1 and camera #2. In this case, the 3D model update unit 22 identifies, for example, the point cloud data generated by camera #2 at substantially the same timing as the timing when the point cloud data is generated by camera #1. The 3D model update unit 22 integrates the point cloud data for each camera generated at substantially the same timing in this way, and generates an integrated reference model.

[0078] Next, the parameter correction unit 24 calculates conversion parameters for converting the integrated reference model (S26). The parameter correction unit 24 calculates, for example, conversion parameters for converting the integrated reference model so as to show the three-dimensional shape of the object at the timing when the point cloud data is generated by camera #1. At this time, the parameter correction unit 24 corrects the conversion parameters so that the vertices after conversion are also consistent with the point cloud data generated by camera #2.

[0079] Next, the 3D data generation unit 25 converts the reference model using the calculated conversion parameters, and generates mesh data using the converted point cloud data (S27).

[0080] Next, the 3D model update unit 22 updates the integrated reference model using the calculated conversion parameters (S27). For example, the 3D model update unit 22 updates the integrated reference model using the point clouds obtained by applying the inverse conversion parameters to the point cloud data generated by each camera.

[0081] As described above, when the analyzer 20 according to the second embodiment converts the reference model into a three-dimensional shape indicated by a plurality of point cloud data sensed at different timings, the analyzer 20 corrects the conversion parameters using the timing difference. In other words, the analyzer 20 can make the converted point cloud data the same as the point cloud data sensed at substantially the same timing by adjusting the amount of change from the reference model converted using the conversion parameters. Thereby, even when the timings at which the plurality of cameras sense are not synchronized, the analyzer 20 can prevent the quality of the three-dimensional data to be displayed on the user terminal 50 from deteriorating.

[0082] FIG. 13 is a block diagram showing a configuration example of the analyzer 10, the analyzer 20, and the user terminal 50 (hereinafter referred to as the analyzer 10 or the like). Referring to FIG. 13, the analyzer 10 or the like includes a network interface 1201, a processor 1202, and a memory 1203. The network interface 1201 may be used to communicate with other network nodes. The network interface 1201 may include, for example, a network interface card (NIC) compliant with the IEEE 802.3 series.

[0083] The processor 1202 reads and executes software (computer program) from the memory 1203, thereby performing the processing of the analysis device 10 and the like described using the flowchart in the above-described embodiment. The processor 1202 may be, for example, a microprocessor, an MPU, or a CPU. The processor 1202 may include a plurality of processors.

[0084] The memory 1203 is composed of a combination of a volatile memory and a non-volatile memory. The memory 1203 may include storage located away from the processor 1202. In this case, the processor 1202 may access the memory 1203 via an I / O (Input / Output) interface (not shown).

[0085] In the example of FIG. 13, the memory 1203 is used to store a group of software modules. The processor 1202 can perform the processing of the analysis device 10 and the like described in the above-described embodiment by reading and executing these groups of software modules from the memory 1203.

[0086] As described with reference to FIG. 13, each of the processors included in the analysis device 10 and the like in the above-described embodiment executes one or more programs including a group of instructions for causing a computer to perform the algorithms described with reference to the drawings.

[0087] The program includes a set of instructions (or software code) that cause a computer to perform one or more of the functions described in the embodiments when loaded into the computer. The program may be stored on a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, the computer-readable medium or tangible storage medium includes random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD), or other memory technologies, CD-ROM, digital versatile disc (DVD), Blu-ray Disc, or other optical disc storage, magnetic cassettes, magnetic tape, magnetic disk storage, or other magnetic storage devices. The program may be transmitted on a transient computer-readable medium or a communication medium. By way of example and not limitation, the transient computer-readable medium or communication medium includes electrical, optical, acoustic, or other forms of propagated signals.

[0088] Note that the present disclosure is not limited to the above embodiments, and can be appropriately changed without departing from the spirit thereof.

Explanation of Reference Numerals

[0089] 10 Analyzer 11 Communication Unit 12 Calculation Unit 13 Correction Unit 14 Generation Unit 20 Analyzer 21 Parameter Calculation Unit 22 3D Model Update Unit 23 Communication Unit 24 Parameter Correction Unit 25 3D Data Generation Unit 30 Camera 31 Camera 32 Camera 33 Camera 40 Access Point Device 50 User Terminal

Claims

1. A communication unit that receives first three-dimensional sensing data representing a result of sensing an object at a first timing from a first sensor, and receives second three-dimensional sensing data representing a result of sensing the object at a second timing from a second sensor installed at a position different from that of the first sensor; A calculation unit that calculates conversion parameters of representative points of a reference model used to convert the reference model indicating the three-dimensional shape of the object into the three-dimensional shape of the object indicated by the first three-dimensional sensing data and the second three-dimensional sensing data; A correction unit that corrects the conversion parameters so as to convert the three-dimensional shape of the object at the first timing based on the difference between the first timing and the second timing; A generation unit that generates three-dimensional data obtained by converting the reference model using the corrected conversion parameters, and includes: The calculation unit is: A first conversion parameter used to convert a first reference model generated using three-dimensional sensing data representing a result sensed by the first sensor into the three-dimensional shape of the object indicated by the first three-dimensional sensing data, and a second reference model generated using three-dimensional sensing data representing a result sensed by the second sensor are calculated, and a second conversion parameter used to convert the second reference model into the three-dimensional shape of the object indicated by the second three-dimensional sensing data; The correction unit is: An analyzer that corrects the second conversion parameter so as to be converted into the three-dimensional shape of the object when the second sensor senses the object at the first timing based on the difference between the first timing and the second timing.

2. A communication unit that receives first three-dimensional sensing data representing a result of sensing an object at a first timing from a first sensor, and receives second three-dimensional sensing data representing a result of sensing the object at a second timing from a second sensor installed at a position different from that of the first sensor; A calculation unit that calculates conversion parameters of representative points of a reference model used to convert the reference model indicating the three-dimensional shape of the object into the three-dimensional shape of the object indicated by the first three-dimensional sensing data and the second three-dimensional sensing data; A correction unit that corrects the conversion parameters so as to convert the three-dimensional shape of the object at the first timing based on the difference between the first timing and the second timing; A generation unit that generates three-dimensional data obtained by converting the reference model using the corrected conversion parameters; The calculation unit calculates conversion parameters used to convert the integrated reference model generated using the three-dimensional sensing data sensed by the first sensor and the three-dimensional sensing data sensed by the second sensor into the three-dimensional shape of the object indicated by the first three-dimensional sensing data and the second three-dimensional sensing data; When the position of a first vertex among a plurality of vertices of the integrated reference model is converted using the conversion parameters, the position of the converted first vertex at the time of the three-dimensional sensing data sensed by the first sensor, and the first vertex among the plurality of vertices included in the first three-dimensional sensing data, the distance to a second vertex that is in the same position in the two-dimensional space, and the position of the converted first vertex at the time of the three-dimensional sensing data sensed by the second sensor, and the first vertex among the plurality of vertices included in the second three-dimensional sensing data, the distance to a third vertex that is in the same position in the two-dimensional space, and calculates the conversion parameters so that the distances are minimized. An analyzer.

3. The generation unit generates integrated three-dimensional data by integrating first three-dimensional data obtained by converting the first reference model using the first conversion parameters and second three-dimensional data obtained by converting the second reference model using the corrected second conversion parameters. The analyzer according to claim 1.

4. The correction unit When the elapsed time from the time indicating the second timing to the time when the second sensor sensed the object to generate the second reference model is T, and the time indicating the difference between the first timing and the second timing is t, T + t is divided by T and the resulting value is multiplied by the rotation amount and the parallel movement amount included in the second conversion parameter to correct the second conversion parameter. The analyzer according to claim 1 or 3.

5. The correction unit Correct the position of the first vertex after transformation to the position of the fourth vertex based on the difference between the first timing and the second timing, and the distance between the position of the first vertex after transformation and the second vertex, and the position of the fourth vertex, and the distance between the position of the fourth vertex and the third vertex corresponding to the first vertex among the plurality of vertices included in the second three-dimensional sensing data, and calculate the conversion parameter so that the distance is minimized. The analysis device according to claim 2.

6. Receive first three-dimensional sensing data representing the result of sensing an object at a first timing from a first sensor, and receive second three-dimensional sensing data representing the result of sensing the object at a second timing from a second sensor installed at a position different from the first sensor. Calculate a first conversion parameter used to convert a first reference model generated using three-dimensional sensing data representing the result sensed by the first sensor into the three-dimensional shape of the object indicated by the first three-dimensional sensing data, and a second conversion parameter used to convert a second reference model generated using three-dimensional sensing data representing the result sensed by the second sensor into the three-dimensional shape of the object indicated by the second three-dimensional sensing data. Based on the difference between the first timing and the second timing, correct the second conversion parameter so that the second sensor senses the object at the first timing and the object is converted into the three-dimensional shape. A data generation method for generating three-dimensional data obtained by converting the reference model using the corrected conversion parameter.

7. Receive first three-dimensional sensing data representing the result of sensing an object at a first timing from a first sensor, and receive second three-dimensional sensing data representing the result of sensing the object at a second timing from a second sensor installed at a position different from the first sensor. When the position of the first vertex among the plurality of vertices of the integrated reference model generated using the three-dimensional sensing data sensed by the first sensor and the three-dimensional sensing data sensed by the second sensor is converted using the conversion parameters used for conversion into the three-dimensional shape of the object indicated by the first three-dimensional sensing data and the second three-dimensional sensing data, the position of the converted first vertex at the time of the three-dimensional sensing data sensed by the first sensor, the first vertex among the plurality of vertices included in the first three-dimensional sensing data, and the distance to the second vertex that is in the same position in the two-dimensional space, and the position of the converted first vertex at the time of the three-dimensional sensing data sensed by the second sensor, and the distance to the third vertex that is in the same position in the two-dimensional space as the first vertex among the plurality of vertices included in the second three-dimensional sensing data, calculate the conversion parameters so that they are minimized, Based on the difference between the first timing and the second timing, correct the conversion parameters so that they are converted into the three-dimensional shape of the object at the first timing, A data generation method for generating three-dimensional data obtained by converting the integrated reference model using the corrected conversion parameters.

8. Receive first three-dimensional sensing data representing the result of sensing an object at a first timing from a first sensor, and receive second three-dimensional sensing data representing the result of sensing the object at a second timing from a second sensor installed at a position different from the first sensor, Calculate a first conversion parameter used to convert a first reference model generated using three-dimensional sensing data representing the result sensed by the first sensor into the three-dimensional shape of the object indicated by the first three-dimensional sensing data, and a second reference model generated using three-dimensional sensing data representing the result sensed by the second sensor, and a second conversion parameter used to convert it into the three-dimensional shape of the object indicated by the second three-dimensional sensing data, Based on the difference between the first timing and the second timing, correct the second conversion parameter so that it is converted into the three-dimensional shape of the object when the second sensor senses the object at the first timing. A program that causes a computer to generate three-dimensional data obtained by converting the reference model using the corrected conversion parameter.

9. Receive first three-dimensional sensing data representing the result of sensing an object at a first timing from a first sensor, and receive second three-dimensional sensing data representing the result of sensing the object at a second timing from a second sensor installed at a position different from the first sensor. When the position of a first vertex among a plurality of vertices of an integrated reference model generated using the three-dimensional sensing data sensed by the first sensor and the three-dimensional sensing data sensed by the second sensor is converted using a conversion parameter used for conversion into the three-dimensional shape of the object indicated by the first three-dimensional sensing data and the second three-dimensional sensing data, the distance between the position of the converted first vertex at the time of the three-dimensional sensing data sensed by the first sensor and a second vertex that is in the same position in the two-dimensional space as the first vertex among the plurality of vertices included in the first three-dimensional sensing data, and the distance between the position of the converted first vertex at the time of the three-dimensional sensing data sensed by the second sensor and a third vertex that is in the same position in the two-dimensional space as the first vertex among the plurality of vertices included in the second three-dimensional sensing data, calculate the conversion parameter so that they are minimized. Based on the difference between the first timing and the second timing, correct the conversion parameter so that it is converted into the three-dimensional shape of the object at the first timing. A program that causes a computer to generate three-dimensional data obtained by converting the integrated reference model using the corrected conversion parameter.

Images