Information processing device

Abstract

When generating a model with a distance measuring sensor, the presence of highly reflective objects, such as mirrors, can prevent accurate distance measurement and thus the modeling of the target space. [Solution] The present invention is an information processing device for generating a spatial model, comprising: a distance measuring unit that acquires distance data to an object; a model generation unit that generates a spatial model using the distance data; and an evaluation unit that evaluates whether or not a symmetrical portion exists from at least one of the distance data and the spatial model, wherein the model generation unit generates the spatial model based on the evaluation result of the evaluation unit.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus for generating a spatial model, and to an information processing apparatus that evaluates acquired distance data or a generated model and generates a model based on the evaluation result.

[0002] In recent years, LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) sensors that measure the distance to an object by irradiating the object with laser light and measuring the round-trip time of the reflected light have become widely popular and have been installed in information terminals such as smartphones and tablet terminals.

[0003] In this information terminal, by scanning an actually existing object with a LiDAR sensor, a three-dimensional computer graphics model or a planar map (hereinafter, these are collectively referred to as a spatial model or simply a model) can be easily generated. The model generated by the information terminal can be viewed from any angle or combined with other images or models, and is widely applied in combination with technologies such as virtual reality (VR), augmented reality (AR), and mixed reality (MR) in addition to the fields of art and design. In such applications, it is required that the target object required for each application be accurately and highly accurately modeled in the material model.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

[0005] This invention solves the problem that when generating a model using an information terminal equipped with a distance measuring sensor such as a LiDAR sensor, the presence of a highly reflective object such as a mirror (hereinafter simply referred to as a mirror) prevents the accurate modeling of the target space. For example, when measuring the distance of a mirror surface with a LiDAR sensor, the emitted laser light is specularly reflected by the mirror surface, and the light that returns directly to the terminal is extremely weak. (However, this is the case when the distance is measured from a direction not perpendicular to the mirror surface.) Most of the laser light is specularly reflected by the mirror surface and is then projected onto some object A. The laser light reflected by the object travels the same optical path, is reflected again by the mirror surface, and returns to the information terminal. In order for the LiDAR sensor to correctly measure the distance of the mirror surface, the laser light should ideally only travel the round-trip distance between the terminal and the mirror surface. However, it will also travel the round-trip distance between the mirror surface and object A, causing the mirror surface to be measured at a longer distance than its true distance. By generating a model based on this erroneous distance data, the information terminal generates an incorrect model that differs from the actual shape of the target space.

[0006] The object of the present invention is to provide an information processing device that prevents the generation of incorrectly shaped models when generating models including mirrored surfaces, and enables the generation of models with correct shapes.

[0007] To achieve the above objective, the present invention provides an information processing device for generating a spatial model, comprising: a distance measuring unit for acquiring distance data to an object; a model generation unit for generating a spatial model using the distance data; and an evaluation unit for evaluating whether a symmetrical portion exists from at least one of the distance data and the spatial model, wherein the model generation unit generates the spatial model based on the evaluation result of the evaluation unit.

[0008] According to the information processing device of the present invention, when generating a model that includes a mirrored surface, it becomes possible to prevent the generation of a model with an incorrect shape and generate a model with a correct shape. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows the hardware configuration of the information processing device according to the first embodiment. [Figure 2] This is a flowchart of the model generation process of the information processing device of the first embodiment. [Figure 3] This figure shows the hardware configuration of the information processing device of the second embodiment. [Figure 4] This is a flowchart of the model generation process of the information processing device in the second embodiment. [Figure 5] This is a mode transition diagram for an information processing device. [Modes for carrying out the invention]

[0010] Two embodiments of the present invention will be described. In the first embodiment, a model is generated from distance data obtained by the distance measuring unit and position data and attitude data obtained by the position and attitude information acquisition unit, and the model is evaluated using only the coordinates that indicate the shape of the model. In the second embodiment, in addition to the first embodiment, an image obtained by the image capturing unit is mapped as a texture to generate a model, and the model is evaluated using the color and features of the texture in addition to the coordinates that indicate the shape of the model.

[0011] [First Embodiment] A first embodiment will now be described. Figure 1 shows the hardware configuration of the information processing device 1 in the first embodiment. The information processing device 1 includes an arithmetic unit 101 such as a CPU (Central Processing Unit), an information processing unit 102 such as an information processing LSI (Large-Scale Integration), a primary storage unit 103 such as a volatile main memory, a secondary storage unit 104 such as a non-volatile auxiliary memory, a distance measuring unit 105 including a laser irradiation device and a light receiving device, a position and orientation information acquisition unit 106 such as a position sensor and an acceleration sensor, an operation unit 107 such as buttons and a touch panel, and a display unit 108 such as a liquid crystal display. Data is exchanged between each component of the information processing device 1 via a bus 110.

[0012] The calculation unit 101 controls other hardware. Each component shown in Figure 1 is controlled by the calculation unit 101, including parameter setting and operation instructions. The information processing unit 102 performs calculation processing on distance data acquired by the distance measuring unit 105 and position and attitude data acquired by the position and attitude information acquisition unit 106, as well as model generation and model evaluation processing. The primary storage unit 103 temporarily stores data used by the calculation unit 101 and the information processing unit 102. The secondary storage unit 104 stores data used by the calculation unit 101 in processing and models generated by the information processing unit 102. The distance measuring unit 105 irradiates the target object with laser light, receives the reflected light, measures the time of reception, and measures the distance to the target object. The position and attitude information acquisition unit 106 measures the position and inclination of the information processing device 1 and acquires position and attitude data. The operation unit 107 inputs instructions from the user to the calculation unit 101. The display unit 108 displays the generated model and presents it to the user.

[0013] Figure 5 shows the mode transition diagram of the information processing device 1. The information processing device 1 has at least a standby mode 501 and a scan mode 502, and may also have a view mode 503. The initial state is standby mode 501. In standby mode 501, the information processing device 1 waits until scanning of the target object begins. In standby mode 501, the user adjusts the angle of the information processing device 1 to point it at the target object to be scanned. After the user has determined the angle of the information processing device 1, when the user presses the button on the operation unit 107 that is responsible for starting the scan, the information processing device 1 transitions to scan mode 502. In scan mode 502, the information processing device 1 performs model generation processing. When the user presses the button on the operation unit 107 that is responsible for stopping the scan, the information processing device 1 transitions to standby mode 501. In standby mode 501, when the user presses the button on the operation unit 107 that is responsible for transitioning to view mode 503, the information processing device 1 transitions to view mode 503. In view mode 503, the display unit 108 of the information processing device 1 displays the generated model according to user input from the operation unit 107, and depending on the input, it displays the model by scaling, rotating, and moving it. In view mode 503, when the user presses the button on the operation unit 107 that is responsible for transitioning to standby mode 501, the information processing device 1 transitions to standby mode 501.

[0014] Next, Figure 2 shows the flow of the model generation process of the information processing device 1 in the first embodiment.

[0015] When the user operates the control unit 107 and switches to scan mode 502, the model generation process is started.

[0016] In Figure 2, the process proceeds to step S201 after the model generation process begins. In step S201, the position and orientation information acquisition unit 106 acquires position data and orientation data of the information processing device 1.

[0017] After that, in step S202, the arithmetic unit 101 records the position data and orientation data acquired by the position and orientation information acquisition unit 106 in the primary storage unit 103.

[0018] After that, in step S203, the distance measurement unit 105 acquires distance data.

[0019] After that, in step S204, the arithmetic unit 101 records the distance data acquired by the distance measurement unit 105 in the primary storage unit 103.

[0020] After that, in step S205, the information processing unit 102 generates a model using the position data, orientation data, and distance data recorded in the primary storage unit 103, and records the generated model data in the primary storage unit 103.

[0021] The model generated by the information processing apparatus 1 is the same as a general computer graphics model. That is, the model is represented by a set of polygon data called a number of polygons. A polygon is composed of vertices, sides connecting two vertices, and faces spanned by three or more sides. The model or each polygon of the model may have material information such as texture, transparency, and reflectivity. Model generation is performed by a known method. That is, the three-dimensional absolute coordinates of vertices, sides, and faces representing the object surface are calculated from the position data and orientation data of the information processing apparatus 1 acquired by the position and orientation information acquisition unit 106, and the distance data acquired by the distance measurement unit 105. In model generation, the vertices constituting the polygon may be uniformly generated on the surface of the target object, or may be intensively generated at characteristic points or characteristic shape parts with large surface curvature. Note that the model representation method and the data structure constituting the model may be other than those described above.

[0022] After that, in step S206, the information processing unit 102 evaluates the generated model.

[0023] The model evaluation in step S206 aims to determine whether there is a plane symmetry part in the model, and uses known methods such as a method using the Hough transform. That is, two vertices constituting the model are arbitrarily extracted, and the relative position and plane direction of the two points are evaluated. The plane direction may be evaluated using not only each of the extracted vertices but also the coordinates of the surrounding vertices, edges, and faces.

[0024] If it is determined that the extracted vertex pair is in a plane-symmetric position by evaluating the relative position and plane direction, the vertex pair is used as a candidate for the symmetric point pair, and the coordinates and angle of the midpoint of the line segment connecting the two points are obtained. Using this information, a voting process is performed in the parameter space representing the equation of the symmetry axis plane. By performing the above processing for all or some of the point pairs in the model and extracting the coordinates with a large cumulative vote count in the parameter space, the symmetry axis plane of the plane-symmetric object existing in the model is obtained.

[0025] After that, in step S207, the information processing unit 102 determines whether there is a plane symmetry part in the generated model. Whether there is a plane symmetry part can be determined by whether there is a value in the parameter space whose vote count in step S206 is greater than a predetermined value.

[0026] If there is a plane symmetry part in the model, the process proceeds to step S208, and if there is no plane symmetry part in the model, the process proceeds to step S209.

[0027] If it is determined in step S207 that there is a plane symmetry part in the model, then in step S208, the information processing unit 102 modifies the model and records the modified model in the primary storage unit 103. The modification of the model deletes one side that is determined to have low accuracy among the parts forming the plane symmetry part of the model. The model accuracy is evaluated by referring to the positions, distributions, connection relationships, etc. of the vertices, edges, and faces forming the polygon. After that, a polygon representing a plane is formed at the position of the symmetry axis plane of the model. If each polygon of the model has material information, etc., characteristics with the characteristics of a mirror may be set for the plane formed here.

[0028] In step S209, following step S208, the display unit 108 displays the model recorded in the primary storage unit 103.

[0029] Subsequently, in step S210, the calculation unit 101 determines whether or not model generation is complete. Model generation is determined to be complete when the user operates the operation unit 107 and the information processing device 1 exits scan mode 502.

[0030] Subsequently, in step S211, the completed model is recorded in the secondary storage unit 104.

[0031] According to the first embodiment described above, the information processing device 1 can correct a model of an incorrect shape that is generated due to the inability to correctly measure the distance of the mirror surface, and generate a model of the correct shape.

[0032] [Second Embodiment] A second embodiment will now be described. Figure 3 shows the hardware configuration of the information processing device 1 in the second embodiment. In addition to the first embodiment, the information processing device 1 in the second embodiment has an image acquisition unit 109 including an image sensor and associated optical devices. The configuration of the second embodiment other than the image acquisition unit 109 is the same as that of the first embodiment. The image acquisition unit 109 collects light from the target object to be photographed, forms an image on the image sensor, and converts this into digital data. In the information processing device 1 of the second embodiment, the calculation unit 101 also controls the image acquisition unit 109. The information processing unit 102 performs calculation processing on the distance data acquired by the distance measuring unit 105, the position data and attitude data acquired by the position and attitude information acquisition unit 106, and the image data captured by the image acquisition unit 109, and uses this distance data, position data, attitude data and image data to perform model generation processing and model evaluation processing. In addition, the information processing unit 102 processes the image captured by the image acquisition unit 109 during model generation so that the display unit 108 can display it as a live view. The display unit 108 displays the model generated by the information processing unit 102, as well as the live view image processed by the information processing unit 102.

[0033] The mode configuration of the information processing device 1 is the same as in the first embodiment. In the second embodiment, in standby mode 501 and scan mode 502, the image captured by the image acquisition unit 109 and processed by the information processing unit 102 may be displayed in live view on the display unit 108. The user can perform the scanning operation while checking the status of scanning the target object by looking at the image displayed in live view.

[0034] Figure 4 shows the flow of the model generation process of the information processing device 1 in the second embodiment. The differences from the first embodiment are that processing steps S405 and S406 related to the image capturing unit 109 are added, the model generation method in step S407 and the model evaluation method in S408 are different, and the display content in step S411 is different.

[0035] The model generation process starts when the user operates the control unit 107 and switches to scan mode 502. In Figure 4, the process proceeds to step S401 after the model generation process has started. The processing from step S401 to step S404 is the same as the processing from step S201 to step S204 in the first embodiment.

[0036] After step S404, in step S405, the image acquisition unit 109 acquires an image of the target object.

[0037] Subsequently, in step S406, the information processing unit 102 processes the image captured by the image capturing unit 109 as a live view image. This processing is similar to the image processing performed by a typical digital camera, including correction of pixel defects and distortion, and development processing.

[0038] Subsequently, in step S407, the information processing unit 102 generates a model using the position data, orientation data, and distance data recorded in the primary storage unit 103, and records the generated model data in the primary storage unit 103. Model generation is performed in the same manner as in the first embodiment. However, in the second embodiment, image data captured by the image acquisition unit 109 may be mapped as a texture.

[0039] Subsequently, in step S408, the information processing unit 102 evaluates the generated model.

[0040] The model evaluation in step S408 aims to determine whether or not there are plane-symmetric areas within the model, similar to the first embodiment. While the model evaluation in the first embodiment focused only on the model shape, i.e., the coordinates of the model components, the second embodiment also utilizes information about the color and features of the image mapped as a texture to the model. Specifically, two vertices constituting the model are arbitrarily extracted, and when evaluating the relative position and face direction of the two points, the color and features of the texture are also evaluated. To evaluate the texture color of a vertex pair, it is sufficient to evaluate whether the color value (color code, etc.) of each vertex falls within a predetermined range. By evaluating the relative position, face direction, and color similarity of the two points, if it is determined that the extracted vertex pair is in a plane-symmetric position, the vertex pair is designated as a candidate for a symmetry point pair. The color and features of the texture-mapped image may be evaluated not only using the coordinates of each extracted vertex, but also using the coordinates of surrounding vertices, edges, and faces, similar to the face direction. Based on this evaluation of relative position, face direction, and texture color and features, the symmetry axis plane of a plane-symmetric object is determined, similar to the first embodiment. By utilizing the color and feature information of the image mapped as the model's texture, the symmetrical structure of the model can be evaluated with greater accuracy.

[0041] Subsequently, in step S409, the information processing unit 102 determines whether or not the generated model has a plane-symmetrical portion. If the model has a plane-symmetrical portion, the process proceeds to step S410; otherwise, the process proceeds to step S411. If it is determined in step S409 that the model has a plane-symmetrical portion, then in step S410, the information processing unit 102 modifies the model and records the modified model in the primary storage unit 103. The model modification is performed in the same manner as in the first embodiment.

[0042] In step S411, following step S410, the display unit 108 displays the live view image processed by the information processing unit 102 and the model generated by the information processing unit 102. In step S411, displaying the model is mandatory, but displaying the live view image is not mandatory. When both the live view image and the model are displayed, the model is displayed by superimposing it on the live view image. The model may be displayed as an opaque or semi-transparent surface model where polygons are represented by surfaces, or as a wireframe model where polygons are represented by edges, or by any other representation method. In this case, it is assumed that the model has not mapped the image captured by the image capturing unit 109 as a texture. When the live view image is not displayed and only the model is displayed, the model is one on which the image captured by the image capturing unit 109 has been mapped as a texture. In either case, the color of the target object represented by the model is superimposed on the model and presented to the display unit 108. This makes it easier for the user to understand which part of the target object is being modeled when scanning the target object. The subsequent steps S412 to S413 are the same as the steps S210 to S211 in the first embodiment.

[0043] According to the second embodiment described above, the information processing device 1 can evaluate the model with higher accuracy than the first embodiment by utilizing the image captured by the image capturing unit 109, correct models with incorrect shapes generated due to the inability to correctly measure the distance of the mirror surface with higher accuracy, and generate models with correct shapes. In addition, the display unit 108 can display to the user which parts of the target object are being modeled and which areas are not being modeled, allowing the user to efficiently perform high-quality model generation work.

[0044] Although the present invention has been described in detail above based on its preferred embodiments, the present invention is not limited to these specific embodiments, and various forms that do not depart from the spirit of the invention are also included in the present invention. Furthermore, each of the embodiments described above is merely one embodiment of the present invention, and it is possible to combine each embodiment as appropriate.

[0045] In this explanation, the model to be generated is assumed to be a 3D model, but the model to be generated and evaluated may also be a 2D model.

[0046] The information processing device 1 may take any form other than those described in this embodiment. The information processing device 1 is envisioned to be any form of computer such as smartphones, tablet terminals, or wearable devices such as head-mounted displays, digital still cameras or digital video cameras, vehicles such as automobiles, aircraft such as drones, or robots for various purposes, and may take the form of these devices themselves or as devices mounted on them, but is not limited to these.

[0047] For example, each component of the hardware configuration shown in the embodiment, such as a drone and a controller, may be divided and mounted on multiple devices, and they may operate in a coordinated manner. Furthermore, the various controls described above in the flowchart may be performed by a single processor or circuit, or multiple processors or circuits may share the processing to control the entire device. [Explanation of symbols]

[0048] 1. Information Processing Device 101 Arithmetic section 102 Information Processing Unit 103 Primary storage 104 Secondary storage 105 Ranging section 106 Position and orientation information acquisition unit 107 Operation section 108 Display section

Claims

1. An information processing device for generating spatial models, A distance measuring unit that acquires distance data to the target object, A model generation unit that generates a spatial model using the aforementioned distance data, The system includes an evaluation unit that evaluates whether or not a symmetrical portion exists based on at least one of the distance data and the spatial model. The information processing device is characterized in that the model generation unit generates a spatial model based on the evaluation results of the evaluation unit.

2. The information processing apparatus according to claim 1, characterized in that the spatial model generated by the model generation unit is a three-dimensional model, and the evaluation unit evaluates whether or not a plane symmetry exists from at least one of the distance data and the spatial model.

3. The information processing device according to 1, wherein the spatial model generated by the model generation unit is a two-dimensional model, and the evaluation unit evaluates whether or not a line-symmetric portion exists based on at least one of the distance data and the spatial model.

4. The information processing apparatus according to any one of claims 1 to 3, characterized in that the model generation unit generates a model by changing at least one of the visual or physical characteristics such as presence or absence, position, shape, size, angle, connection relationship, color, material, or texture of at least one of the vertices, edges or lines or faces that form the model, based on the evaluation results of the evaluation unit.

5. The information processing apparatus according to any one of claims 1 to 4, further comprising an imaging unit for capturing image data of an object, wherein the evaluation unit evaluates at least one of the distance data, the spatial model, and the image data.

6. The information processing apparatus according to any one of claims 1 to 5, characterized in that the evaluation unit evaluates each data after both distance measurement by the distance measuring unit and imaging by the imaging unit, or after the generation of a model by the model generation unit has been completed.

7. The information processing apparatus according to any one of claims 1 to 5, characterized in that the evaluation unit evaluates each data before at least one of the distance measurement by the distance measuring unit, the imaging by the imaging unit, and the model generation by the model generation unit is completed.

8. The information processing apparatus according to any one of claims 1 to 7, characterized in that the model generation unit generates at least one of vertices, edges, lines, or faces on the axis of symmetry.

9. The information processing apparatus according to any one of claims 1 to 8, characterized in that the model generation unit deletes one of the symmetrical parts of the model that is evaluated as having low accuracy.

10. The information processing apparatus according to claim 9, characterized in that the model generation unit evaluates the model accuracy based on the number of vertices, edges, and faces per unit volume that form the model, the uniformity of the density of vertices, edges, and faces that form the model, the connection relationships between the vertices, edges, and faces that form the model and neighboring elements, and the texture image to be mapped.

Images