PROGRAM, INFORMATION PROCESSING APPARATUS AND INFORMATION PROCESSING METHOD

Abstract

To generate target object information representing the position and orientation of a target object model with high accuracy.SOLUTION: A program according to an embodiment causes an information processing apparatus to function as a coordinate system specifying unit, an area detection unit, an extraction unit, and a generation unit. The coordinate system specifying unit specifies a reference coordinate system representing a reference of a three-dimensional position. The area detection unit detects a two-dimensional target area in which a designated target object is included, in image data. The extraction unit extracts, from three-dimensional point cloud data, extraction point cloud data representing a three-dimensional position of an object included in the target area. The generation unit generates target object information. In the designated target object, a shape of a first portion is defined, and an orientation of the first portion is defined with respect to at least one of coordinate axes in the reference coordinate system. The generation unit generates the target object information by fitting a target object model to the extraction point cloud data under a condition that an orientation of the target object model matches an orientation defined for the first portion of the designated target object.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a program, an information processing device, and an information processing method. [Background technology]

[0002] Systems that automatically perform infrastructure maintenance and inspection, and systems that automatically drive mobile objects such as automobiles, drones, and automated guided vehicles, utilize technology for modeling the surrounding environment. Man-made structures such as road signs, billboards, and buildings are often composed of flat or curved surfaces. Therefore, modeling of the surrounding environment is performed by, for example, placing object models such as flat, curved, and cylindrical objects in a virtual space that virtualizes real space based on three-dimensional point cloud data generated from image data or three-dimensional point cloud data measured by a laser sensor, radar sensor, or the like. Such modeling of the surrounding environment requires technology that can accurately fit object models such as flat, curved, and cylindrical objects to the three-dimensional point cloud data.

[0003] A conventional method for fitting a planar model representing a specific surface of an object to be recognized to three-dimensional point cloud data involves extracting partial point cloud data representing the object to be recognized from the three-dimensional point cloud data, and calculating the surface that has the smallest distance from the extracted partial point cloud data using the least squares method or principal component analysis.

[0004] However, in such conventional methods, when using 3D point cloud data with relatively large errors, there is a high possibility that the calculated position and orientation of the surface will deviate from the actual surface. For example, when 3D point cloud data is calculated by matching pixels of the same object in multiple image data, or when 3D point cloud data is estimated from one or more image data using a neural network, there is a tendency for errors to be larger than when 3D point cloud data is detected using a laser sensor or radar sensor. Therefore, when using 3D point cloud data generated from image data, there is a high possibility that the calculated position and orientation of the surface will deviate from the actual surface.

[0005] Furthermore, if the recognition target is located outdoors and other objects, such as trees or pedestrians, are present in front of the recognition target, occlusion, in which the other objects obscure part of the recognition target, may occur. When occlusion occurs, conventional methods extract partial point cloud data from the 3D point cloud data, including point clouds representing the other objects. When the partial point cloud data including point clouds representing the other objects is extracted, the position and orientation of the calculated surface may deviate from the actual surface due to the influence of the point cloud representing the other objects. For example, a technique is known that reduces the deviation between the calculated surface and the actual surface by decreasing the contribution rate of each of the multiple points included in the 3D point cloud data in calculating the surface as the point deviates from the median, and a technique is known that reduces the deviation between the calculated surface and the actual surface by calculating the surface using robust estimation. However, even with these techniques, when a large number of point clouds representing other objects are included, conventional methods still result in a large deviation between the calculated surface and the actual surface.

[0006] In this way, in conventional methods of fitting an object model to three-dimensional point cloud data, if the three-dimensional point cloud data contains large errors or if occlusion occurs, the position and orientation of the object model will deviate from the actual position and orientation of the object to be recognized. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Patent No. 6829626 Summary of the Invention [Problem to be solved by the invention]

[0008] The problem to be solved by the present invention is to provide a program, an information processing device, and an information processing method for generating object information representing the position and orientation of an object model with high accuracy. [Means for solving the problem]

[0009] A program according to an embodiment causes a computer to function as an information processing device. The program causes the information processing device to function as an image acquisition unit, a point cloud acquisition unit, a coordinate system identification unit, a region detection unit, an extraction unit, a model acquisition unit, a generation unit, and an output unit. The image acquisition unit acquires image data. The point cloud acquisition unit acquires three-dimensional point cloud data including a plurality of three-dimensional points, each representing a three-dimensional position of an object included in the image data. The coordinate system identification unit identifies a reference coordinate system representing a reference for the three-dimensional position. The region detection unit detects a two-dimensional target region in the image data that includes a designated object. The extraction unit extracts extracted point cloud data representing the three-dimensional position of an object included in the target region from the three-dimensional point cloud data. The model acquisition unit acquires an object model, which is information that models the shape of at least a first portion of the designated object. The generation unit generates object information that represents the position and orientation of the object model when the object model is placed in three-dimensional space in correspondence with the designated object. The output unit outputs the object information. The specified object has a defined shape of the first portion and a defined orientation of the first portion relative to at least one coordinate axis in the reference coordinate system. The generation unit generates the object information by fitting the object model to the extracted point cloud data under conditions that match the orientation of the object model to the orientation defined for the first portion of the specified object. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 2 is a diagram showing a vehicle equipped with an information processing device and a designated object. [Figure 2] FIG. 2 is a diagram showing a processing flow by an information processing device. [Figure 3] FIG. 1 is a diagram showing the functional configuration of an information processing apparatus according to a first embodiment. [Figure 4] FIG. 2 is a diagram for explaining a reference coordinate system according to the first embodiment. [Figure 5] FIG. 10 is a diagram for explaining a reference coordinate system according to the second embodiment. [Figure 6] FIG. 10 is a diagram showing the functional configuration of an information processing apparatus according to a third embodiment. [Figure 7] FIG. 11 is a diagram for explaining a reference coordinate system according to the third embodiment. [Figure 8] 10 is a flowchart showing the flow of processing by a generation unit according to the fourth embodiment. [Figure 9] FIG. 10 is a diagram showing a first example of a method for calculating an estimated plane. [Figure 10] FIG. 10 is a diagram showing a second example of a method for calculating an estimated plane. [Figure 11] FIG. 10 is a diagram showing a transformation vector from a model coordinate system to a reference coordinate system. [Figure 12] FIG. 1 is a diagram showing an object model represented by a three-dimensional model. [Figure 13] FIG. 13 is a diagram showing the functional configuration of an information processing device according to a sixth embodiment. [Figure 14] FIG. 13 is a diagram showing the functional configuration of an information processing apparatus according to a seventh embodiment. [Figure 15] FIG. [Figure 16] FIG. 1 is a diagram showing an example of a hardware configuration of an information processing apparatus. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, several embodiments of the present invention will be described with reference to the drawings.

[0012] (First embodiment) FIG. 1 is a diagram showing a vehicle 200 equipped with an information processing device 20 according to the first embodiment and a designated object 100. As shown in FIG.

[0013] The information processing device 20 according to the first embodiment generates object information representing the position and posture of an object model (e.g., information representing a plane) that represents the shape of a first part (e.g., a main surface on which information such as a road sign is depicted), which is at least a part of a specified object 100 (e.g., a road sign board, a signboard, etc.) that exists in real space, when the object model is placed in three-dimensional space in correspondence with the specified object 100.

[0014] For example, the information processing device 20 is mounted on a vehicle 200, which is an example of a moving body. The vehicle 200 includes a camera 210. The camera 210 captures images of objects around the vehicle 200 to generate image data. The vehicle 200 may further include a three-dimensional sensor device 220 such as a laser sensor or a radar sensor. The three-dimensional sensor device 220 generates three-dimensional point cloud data indicating the three-dimensional positions of objects around the vehicle 200.

[0015] Vehicle 200 further includes control device 230. Control device 230 controls the traveling of vehicle 200, assists the driver on board vehicle 200 in driving vehicle 200, accepts input from the driver or the like, and displays information to the driver or the like.

[0016] The information processing device 20 acquires image data captured by the camera 210. The information processing device 20 may acquire three-dimensional point cloud data from the three-dimensional sensor device 220. The information processing device 20 generates object information based on the acquired data, and provides the generated object information to a control device 230 that controls the vehicle 200. Based on the acquired object information, the control device 230 recognizes information indicated on the designated object 100 and controls the vehicle 200, or displays information about the designated object 100 to the driver or the like.

[0017] The information processing device 20 may be provided outside the vehicle 200. The information processing device 20 may also be mounted on a moving body other than the vehicle 200. For example, the information processing device 20 may be mounted on a moving body such as a drone or an automatic guided vehicle.

[0018] FIG. 2 is a diagram showing the flow of processing by the information processing device 20.

[0019] First, the information processing device 20 acquires image data captured by the camera 210 (S11). In addition, the information processing device 20 acquires, together with the image data, three-dimensional point cloud data including a plurality of three-dimensional points each representing a three-dimensional position of an object included in the image data (S12).

[0020] Next, the information processing device 20 identifies a reference coordinate system that represents the reference for the three-dimensional position of the space in which the designated object 100 exists (S13).

[0021] The reference coordinate system is represented by an origin and three coordinate axes. The three coordinate axes are represented by, for example, mutually orthogonal x-axis, y-axis, and z-axis. A three-dimensional position represented by the reference coordinate system is represented by the distance from the origin in each of the x-axis, y-axis, and z-axis directions.

[0022] In the first embodiment, the reference coordinate system is a camera coordinate system based on the camera 210 that generates the image data. The camera coordinate system will be described in detail below with reference to FIG.

[0023] Next, the information processing device 20 detects a two-dimensional target area in the image data that includes the specified target object 100 (S14).

[0024] The designated object 100 is an object that exists in real space. The shape of at least a part of the designated object 100, a first portion, is defined. Furthermore, the orientation of the first portion of the designated object 100 is defined with respect to at least one coordinate axis in a reference coordinate system. The orientation of the first portion of the designated object 100 may be defined with respect to a specific coordinate axis among three coordinate axes in the reference coordinate system. Furthermore, the coordinate axis defining the orientation of the first portion of the designated object 100 may be unknown. In other words, it is unknown which of the three coordinate axes in the reference coordinate system the first portion of the designated object 100 corresponds to, but the orientation of the first portion of the designated object 100 with respect to any of the three coordinate axes may be defined.

[0025] The first portion of the designated object 100 may be, for example, the surface of a rectangle or a cube, or the side surface of a cylinder. The first portion of the designated object 100 may also be the entire designated object 100.

[0026] The target region may have any shape as long as it is a two-dimensional region containing one or more pixels in the image data. Furthermore, the target region may include other objects besides the designated object 100, as long as it includes at least the designated object 100. Furthermore, the target region may include an area where part of the designated object 100 is hidden by another object due to occlusion or the like. For example, if the designated object 100 is a road sign, the target region may include trees or pedestrians that are located on the camera 210 side of the road sign.

[0027] Next, the information processing device 20 extracts extracted point cloud data representing the three-dimensional positions of objects included in the target region from the three-dimensional point cloud data (S15).

[0028] Each of the multiple 3D points included in the 3D point cloud data is associated with a pixel in the image data. That is, each of the multiple 3D points included in the 3D point cloud data is associated with a position in the image data at which it is displayed. Therefore, the information processing device 20 can extract one or more corresponding 3D points included in the 3D point cloud data by identifying a partial area in the image data.

[0029] Next, the information processing device 20 generates object information that indicates the position and orientation of the object model when the object model is placed in three-dimensional space in correspondence with the specified object 100 (S16).

[0030] The object model is information that models the shape of a first portion of the specified object 100. For example, the object model may be an equation that represents the shape. For example, the object model may be an equation described using three variables that represent values ​​of three coordinate axes in a reference coordinate system. Furthermore, the object model may be a two-dimensional model or a three-dimensional model that represents a three-dimensional shape using a model coordinate system that is different from the reference coordinate system.

[0031] When generating the object information, the information processing device 20 generates the object information by fitting the object model to the extracted point cloud data under the condition that the orientation of the object model matches the orientation defined for the first part of the specified object 100. For example, if the specified object 100 is defined as being parallel or perpendicular to a predetermined direction in the reference coordinate system, the information processing device 20 makes the object model parallel or perpendicular to the predetermined direction and fits the object model to the extracted point cloud data.

[0032] Then, the information processing device 20 outputs object information that represents the position and orientation of the object model fitted to the extracted point cloud data (S17).

[0033] 2A shows the extracted point cloud data and the object model fitted to the extracted point cloud data in a case where the designated object 100 is a road sign board, the first portion of the designated object 100 is the main surface of the road sign board, and the object model is information representing a plane. In FIG. 2A, the reference coordinate system is the camera coordinate system. FIG. 2A also shows the extracted point cloud data and the object model viewed from above the y-axis.

[0034] FIG. 3 is a diagram showing the functional configuration of the information processing device 20 according to the first embodiment.

[0035] The information processing device 20 includes an image acquisition unit 32, a point cloud acquisition unit 34, a coordinate system identification unit 36, an object setting unit 38, an area detection unit 40, an extraction unit 42, a model storage unit 44, a model acquisition unit 46, a generation unit 48, and an output unit 50.

[0036] The image acquisition unit 32 acquires image data captured by the camera 210. The image data may be a grayscale image or a color image. The camera 210 may be a monocular camera or a stereo camera.

[0037] The point cloud acquisition unit 34 acquires three-dimensional point cloud data including a plurality of three-dimensional points each representing a three-dimensional position of an object included in the image data.

[0038] The three-dimensional point cloud data is sensor data detected by the three-dimensional sensor device 220. The three-dimensional point cloud data may be generated based on a single image data or multiple time-series image data captured by the camera 210. For example, the three-dimensional point cloud data may be generated based on the image data using Visual SLAM (Simultaneous Localization and Mapping), or may be generated using a Structure from Motion technique. For example, the three-dimensional point cloud data may be generated using a neural network that estimates the depth of each pixel position in the image data, or may be generated using another three-dimensional reconstruction technique that creates a three-dimensional representation of an object included in the image data.

[0039] The coordinate system specification unit 36 ​​specifies a reference coordinate system that indicates the reference of the three-dimensional position of the space in which the specified target object 100 exists. In the first embodiment, the coordinate system specification unit 36 ​​specifies a camera coordinate system based on the camera 210 that captured the image data as the reference coordinate system. The camera coordinate system will be described in detail later with reference to FIG. 4.

[0040] The object setting unit 38 sets the designated object 100. For example, the object setting unit 38 may set one object registered in advance as the designated object 100. Alternatively, the object setting unit 38 may set an object selected by the user, for example, from among a plurality of objects registered in advance, as the designated object 100.

[0041] The area detection unit 40 acquires image data from the image acquisition unit 32. Furthermore, the area detection unit 40 acquires information for identifying the designated object 100 from the object setting unit 38. The area detection unit 40 detects a two-dimensional object area that includes the designated object 100 in the acquired image data.

[0042] The region detection unit 40 may receive an operation from a user and detect a region designated by the user as a target region. The region detection unit 40 may detect a rectangular region identified as a designated object 100 using an object recognition technique as a target region. The region detection unit 40 may also detect a region including pixels identified as a designated object 100 using semantic segmentation, instance segmentation, or other identification techniques as a target region.

[0043] The extraction unit 42 acquires three-dimensional point cloud data from the point cloud acquisition unit 34. The extraction unit 42 also acquires a target region from the region detection unit 40. The extraction unit 42 extracts extracted point cloud data representing the three-dimensional positions of objects included in the target region from the three-dimensional point cloud data. Note that the extracted point cloud data may include three-dimensional points representing objects other than the specified target object 100.

[0044] The model storage unit 44 stores a set of an object model and regulating information. For example, when one object selected from a plurality of pre-registered objects is set as the designated object 100, the model storage unit 44 stores a set of an object model and regulating information for each of the plurality of pre-registered objects.

[0045] The definition information is information that represents the orientation of the first portion of the specified object 100, the shape of which is defined, relative to at least one coordinate axis in the reference coordinate system.

[0046] For example, assume that the information processing device 20 is mounted on a vehicle 200, the designated object 100 is a road sign board, the main surface on which the road sign is drawn is the first portion, and the first portion is provided facing the lane of the road. In this case, for example, the model storage unit 44 stores information representing a plane as the object model. Also, in this case, for example, the model storage unit 44 stores, as the regulating information, information indicating that the orientation of the first portion is perpendicular to a coordinate axis parallel to the optical axis of the camera 210 in the reference coordinate system (camera coordinate system in the first embodiment).

[0047] Also, for example, assume that the information processing device 20 is mounted on a vehicle 200, the designated object 100 is a signboard on a pedestrian walkway on the side of a road or on a building on the side of the road, the main surface on which information is written is the first portion, and the first portion is arranged parallel to the lane of the road. In this case, the model storage unit 44 stores information representing the plane as the object model. In addition, in this case, the model storage unit 44 stores, as stipulating information, information indicating that the orientation of the first portion is parallel to a coordinate axis parallel to the optical axis of the camera 210 in the reference coordinate system (the camera coordinate system in the first embodiment) and is parallel to a coordinate axis parallel to a direction corresponding to the vertical direction in the image data.

[0048] Also, for example, assume that information processing device 20 is mounted on vehicle 200, a signboard on the side wall of a tunnel is the specified object 100, the main surface on which information is written is the first portion, and the first portion is the shape of a portion of the inner surface of a hollow cylinder having a central axis parallel to the lanes of a road. In this case, for example, model storage unit 44 stores information representing the inner curved surface of the hollow cylinder as the object model. Also, in this case, model storage unit 44 stores, as stipulating information, information indicating that the central axis of the hollow cylinder forming the first portion is parallel to a coordinate axis parallel to the optical axis of camera 210 in the reference coordinate system (camera coordinate system in the first embodiment).

[0049] The model acquisition unit 46 acquires information identifying the designated object 100 from the object setting unit 38. The model acquisition unit 46 acquires, from the model storage unit 44, a set of an object model and regulating information corresponding to the set designated object 100.

[0050] The generation unit 48 acquires the extracted point cloud data from the extraction unit 42. The generation unit 48 also acquires the reference coordinate system from the coordinate system identification unit 36. Furthermore, the generation unit 48 acquires a set of the object model and the regulating information from the model acquisition unit 46.

[0051] Based on the extracted point cloud data, the reference coordinate system, the object model, and the specification information, the generation unit 48 generates object information that represents the position and orientation of the object model when the object model is placed in three-dimensional space in correspondence with the specified object 100. In this case, the generation unit 48 generates the object information by fitting the object model to the extracted point cloud data under the condition that the orientation of the object model matches the orientation specified for the first part of the specified object 100.

[0052] For example, the generation unit 48 places the object model in the reference coordinate system by adjusting the position and orientation of the object model so as to minimize the distance from the extracted point cloud data represented by the reference coordinate system, under the condition that the orientation of the object model matches the orientation specified for the first part of the specified object 100.

[0053] For example, assume that the information processing device 20 is mounted on a vehicle 200, the designated object 100 is a road sign board, the main surface on which the road sign is drawn is the first portion, and the first portion is provided facing the lane of the road. Furthermore, assume that the object model is information representing a plane. In this case, the generation unit 48 fixes the orientation of the object model perpendicular to a coordinate axis parallel to the optical axis of the camera 210 in the reference coordinate system (the camera coordinate system in the first embodiment), and places the object model in three-dimensional space so as to minimize the distance from the extracted point cloud data. The generation unit 48 then generates object information representing the position and orientation of the object model placed so as to minimize the distance from the extracted point cloud data.

[0054] Also, for example, assume that the information processing device 20 is mounted on a vehicle 200, the designated object 100 is a signboard on a pedestrian walkway or a building on the side of a road, the main surface on which information is written is a first portion, and the first portion is arranged parallel to the lane of the road. Furthermore, assume that the object model is information representing a plane. In this case, the generation unit 48 fixes the orientation of the object model parallel to a coordinate axis parallel to the optical axis of the camera 210 in the reference coordinate system (the camera coordinate system in the first embodiment) and parallel to a coordinate axis parallel to a direction corresponding to the vertical direction in the image data, and places the object model in three-dimensional space so as to minimize the distance from the extracted point cloud data. The generation unit 48 then generates object information representing the position and orientation of the object model arranged so as to minimize the distance from the extracted point cloud data.

[0055] Also, for example, assume that the information processing device 20 is mounted on a vehicle 200, the specified object 100 is a signboard on the side wall of a tunnel, the main surface on which information is written is a first portion, and the first portion is the shape of a portion of the inner curved surface of a hollow cylinder having a central axis parallel to the lanes of a road. Furthermore, assume that the object model is information representing the inner curved surface of the hollow cylinder. In this case, the generation unit 48 fixes the central axis of the object model parallel to a coordinate axis parallel to the optical axis of the camera 210 in the reference coordinate system (the camera coordinate system in the first embodiment), and places the object model in three-dimensional space so as to minimize the distance from the extracted point cloud data. Then, the generation unit 48 generates object information representing the position and orientation of the object model placed so as to minimize the distance from the extracted point cloud data.

[0056] The output unit 50 acquires the object information from the generation unit 48. The output unit 50 outputs the acquired object information to the control device 230. Then, the control device 230 recognizes the information indicated in the designated object 100 and controls the vehicle 200, or displays information about the designated object 100 generated based on the object information to the user.

[0057] The information processing device 20 as described above generates object information that represents the position and orientation of an object model (e.g., information representing a plane or information representing the inner curved surface of a hollow cylinder) that represents the shape of a first portion (main surface) that is at least a part of a specified object 100 (e.g., a road sign board or a billboard) when the object model is placed in three-dimensional space in correspondence with the specified object 100. In particular, the orientation of the first portion of the specified object 100 is defined with respect to at least one coordinate axis in a reference coordinate system. The information processing device 20 then generates the object information by fitting the object model to the extracted point cloud data under the condition that the orientation of the object model matches the orientation defined for the first portion of the specified object 100.

[0058] As a result, the information processing device 20 can accurately generate object information that represents the position and orientation of the object model when it is placed in three-dimensional space in correspondence with the specified object 100. For example, the information processing device 20 can accurately generate object information even when the three-dimensional point cloud data contains a large error or when occlusion occurs.

[0059] FIG. 4 is a diagram for explaining the camera coordinate system.

[0060] In the first embodiment, the coordinate system specification unit 36 ​​specifies, as the reference coordinate system, a camera coordinate system based on the camera 210 that captured the image data. More specifically, in the first embodiment, the reference coordinate system has the three-dimensional position of the camera 210 as its origin, the x-axis as the direction corresponding to the horizontal direction in the image data captured by the camera 210, the y-axis as the direction corresponding to the vertical direction in the image data captured by the camera 210, and the z-axis as the direction parallel to the optical axis of the camera 210. However, the x-axis, y-axis, and z-axis may be assigned in any manner as long as they are assigned without overlapping with any one of the direction corresponding to the horizontal direction in the image data, the direction corresponding to the vertical direction in the image data, and the direction parallel to the optical axis of the camera 210.

[0061] When the image acquisition unit 32 acquires a plurality of image data captured in time series, the coordinate system identification unit 36 ​​identifies the camera coordinate system of any image data among the plurality of image data as the reference coordinate system. The coordinate system identification unit 36 ​​acquires image data in which the size of the included designated object 100 is larger than the reference size, image data in which the entire designated object 100 is included, image data captured while the camera 210 is moving linearly, or image data selected at random, and acquires the reference coordinate system based on the acquired image data.

[0062] Fig. 4A shows the relationship between the camera position, extracted point cloud data, and reference coordinate system when a vehicle 200 equipped with an information processing device 20 passes near a road sign board, which is a designated object 100. The extracted point cloud data in Fig. 4A represents three-dimensional points viewed from above the y-axis. The coordinate system identification unit 36 ​​identifies the reference coordinate system based on any one of the image data of multiple image data captured continuously by a moving camera 210.

[0063] When the information processing device 20 is mounted on the vehicle 200, the optical axis of the camera 210 is often parallel to the traveling direction of the vehicle 200. Furthermore, artificial objects such as road signs or billboards that are installed near the road are often installed so that their planar portions are perpendicular or parallel to the traveling direction of the vehicle 200. For this reason, when the camera coordinate system is used as the reference coordinate system and an artificial object installed near the road is used as the designated object 100, the information processing device 20 can easily match the orientation of the object model to the orientation defined for the first portion of the designated object 100.

[0064] In this way, according to the information processing device 20, by appropriately selecting the specified object 100 and the reference coordinate system, the orientation of the object model can be easily matched to the orientation defined for the first part of the specified object 100, and as a result, object information can be generated with a small amount of calculation.

[0065] (Second embodiment) Next, an information processing device 20 according to a second embodiment will be described.

[0066] The information processing device 20 according to the second embodiment has substantially the same configuration and functions as the information processing device 20 according to the first embodiment. In the following description of the information processing device 20 according to the second embodiment, differences will be described, and components having substantially the same functions as those in the first embodiment will be assigned the same reference numerals and detailed description will be omitted. The same applies to the third and subsequent embodiments.

[0067] FIG. 5 is a diagram illustrating a coordinate system in which one of the three coordinate axes is parallel to the lane markings of the road.

[0068] In the second embodiment, the information processing device 20 is mounted on a vehicle 200. In the second embodiment, the information processing device 20 sets an artificial object provided around a road as the designated object 100.

[0069] In the second embodiment, the coordinate system specification unit 36 ​​specifies a coordinate system in which one of the three coordinate axes is parallel to a road dividing line as the reference coordinate system. More specifically, in the second embodiment, the reference coordinate system has an origin at an arbitrary point on a first dividing line, which is an arbitrary dividing line on the road, an x-axis that is parallel to the road plane and perpendicular to the first dividing line, a y-axis that is perpendicular to the road plane, and a z-axis that is parallel to the first dividing line.

[0070] However, the origin may be any three-dimensional position. The x-axis, y-axis, and z-axis may be assigned in any way as long as they are assigned to one of the directions parallel to the road plane and perpendicular to the first lane line, the direction perpendicular to the road plane, and the direction parallel to the first lane line without overlapping.

[0071] The lane markings are lines drawn on the road to guide the vehicle 200. For example, lane markings include a center line drawn in the approximate center of the road and outer roadside lines that define the outside of the road.

[0072] In the second embodiment, the coordinate system identification unit 36 ​​identifies such a reference coordinate system based on, for example, image data or previously acquired map information. For example, the coordinate system identification unit 36 ​​acquires a region of lane lines from the image data using an identification technique such as semantic segmentation or instance segmentation. The coordinate system identification unit 36 ​​then maps the lane lines included in the acquired region into three-dimensional space and identifies the reference coordinate system by fitting straight line data, curve data, or the like to the lane lines mapped into three-dimensional space.

[0073] Fig. 5A shows the relationship between the camera position, extracted point cloud data, and the reference coordinate system when a vehicle 200 passes near a road sign board, which is the designated object 100. The extracted point cloud data in Fig. 5A represents three-dimensional points viewed from above on the y-axis.

[0074] Artifacts such as road signs or billboards installed near roads are often installed so that their planar portions are perpendicular or parallel to the direction of the road's dividing lines. Therefore, when a coordinate system in which one of the three coordinate axes is parallel to the road's dividing lines is used as the reference coordinate system and an artefact installed near a road is set as the designated object 100, the information processing device 20 can easily match the orientation of the object model to the orientation defined for the first part of the designated object 100. Therefore, the information processing device 20 according to the second embodiment can generate object information with a small amount of calculation.

[0075] (Third embodiment) Next, an information processing device 20 according to a third embodiment will be described.

[0076] FIG. 6 is a diagram showing the functional configuration of an information processing device 20 according to the third embodiment.

[0077] In the third embodiment, the information processing device 20 is mounted on a vehicle 200. In the third embodiment, the information processing device 20 sets an artificial object provided around a road as the designated object 100.

[0078] The information processing device 20 further includes a motion detection section 60 in addition to the configuration of the information processing device 20 according to the first embodiment.

[0079] The motion detection unit 60 detects the motion of the camera 210. More specifically, the motion detection unit 60 detects parameters that represent the translational and rotational movements of the camera 210 over time. For example, the motion detection unit 60 may detect the motion of the camera 210 using Visual SLAM based on two or more pieces of image data captured by the camera 210 in time series, or may detect the motion of the camera 210 using a Structure from Motion technique. Furthermore, for example, the motion detection unit 60 may detect the motion of the camera 210 using a neural network that estimates the motion of the camera 210 from the image data. Furthermore, the motion detection unit 60 may detect the motion of the camera 210 based on sensor data acquired by sensors other than the camera 210, such as a global positioning system (GPS), an acceleration sensor, and an angular velocity sensor, or may detect the motion of the camera 210 by combining these sensor data.

[0080] In the third embodiment, the coordinate system identification unit 36 ​​acquires the movement of the camera 210 detected by the movement detection unit 60. The coordinate system identification unit 36 ​​calculates the movement direction of the camera 210 based on the movement of the camera 210. Then, the coordinate system identification unit 36 ​​identifies a reference coordinate system based on the calculated movement direction of the camera 210.

[0081] FIG. 7 is a diagram for explaining a coordinate system in which one of the three coordinate axes is parallel to the movement direction of the camera 210. In FIG.

[0082] In the third embodiment, the coordinate system specification unit 36 ​​specifies, as the reference coordinate system, a coordinate system in which one of the three coordinate axes is parallel to the movement direction of the camera 210. More specifically, in the third embodiment, the reference coordinate system has the three-dimensional position of the camera 210 as its origin, the x-axis as a direction perpendicular to the movement direction of the camera 210 and parallel to the road, the y-axis as a direction perpendicular to the movement direction of the camera 210 and perpendicular to the road, and the z-axis as a direction parallel to the movement direction of the camera 210.

[0083] However, the x-axis, y-axis, and z-axis may be assigned in any manner as long as they are assigned to one of the following directions without overlap: a direction perpendicular to the direction of movement of camera 210 and parallel to the road; a direction perpendicular to the direction of movement of camera 210 and perpendicular to the road; and a direction parallel to the direction of movement of camera 210.

[0084] The direction of movement of the camera 210 is the direction of change in the three-dimensional position of any point on the camera 210.

[0085] 7A shows the relationship between the camera position and imaging direction, extracted point cloud data, movement direction of the camera 210, and the reference coordinate system when a vehicle 200 equipped with an information processing device 20 passes near a road sign board, which is the designated object 100. The extracted point cloud data in FIG. 7A represents three-dimensional points viewed from above on the y-axis.

[0086] After purchasing the vehicle 200, the user may attach, for example, a drive recorder to the vehicle 200. In such a case, the direction of the optical axis of the camera 210 may deviate from the traveling direction of the vehicle 200. If the direction of the optical axis of the camera 210 deviates from the traveling direction of the vehicle 200, the optical axis of the camera 210 will not be perpendicular to the main surface of the road sign board. Furthermore, if the direction of the optical axis of the camera 210 deviates from the traveling direction of the vehicle 200, the optical axis of the camera 210 will not be parallel to the central axis of a pedestrian walkway on the side of the road, a signboard on a building on the side of the road, or a tube forming a tunnel. For this reason, when an artificial object installed around the road is set as the designated object 100, the information processing device 20 cannot easily perform a process of matching the orientation of the object model with the orientation defined for the first part of the designated object 100, even if it specifies the camera coordinate system as the reference coordinate system.

[0087] In contrast to this, the information processing device 20 according to the third embodiment uses a coordinate system in which one of the three coordinate axes is parallel to the movement direction of the camera 210 as the reference coordinate system.

[0088] When the information processing device 20 is mounted on a vehicle 200, the movement direction of the camera 210 is parallel to the traveling direction of the vehicle 200. Furthermore, artificial objects such as road signs or billboards that are installed near roads are often installed so that their planar portions are perpendicular or parallel to the traveling direction of the vehicle 200. For this reason, when a coordinate system in which one of the three coordinate axes is parallel to the traveling direction of the camera 210 is used as the reference coordinate system and an artificial object installed near the road is set as the designated object 100, the information processing device 20 can easily match the orientation of the object model to the orientation defined for the first part of the designated object 100. As a result, the information processing device 20 according to the third embodiment can generate object information with a small amount of calculation.

[0089] (Fourth embodiment) Next, an information processing device 20 according to a fourth embodiment will be described.

[0090] In the fourth embodiment, the first portion of the designated object 100 is a plane. Also, in the fourth embodiment, the object model is an equation of a plane.

[0091] In the fourth embodiment, the reference coordinate system may be the camera coordinate system used in the first embodiment, the coordinate system used in the second embodiment in which one of the three coordinate axes is parallel to a road dividing line, or the coordinate system used in the third embodiment in which one of the three coordinate axes is parallel to the movement direction of the camera 210. However, the orientation of the first portion of the designated object 100 is defined as being parallel to any one of the three coordinate axes in the reference coordinate system. For example, if the information processing device 20 is mounted on a vehicle 200 and the designated object 100 is an artificial object provided around a road, such as a road sign or billboard, the orientation of the first portion of the designated object 100 is defined as being parallel to or perpendicular to the optical axis direction of the camera 210, the road dividing line, or the movement direction of the camera 210.

[0092] In the fourth embodiment, the generation unit 48 constrains the plane represented by the equation to be parallel to a coordinate axis parallel to the first portion of the reference coordinate system, and calculates an estimated plane that minimizes the distance from each of the multiple 3D points included in the extracted point cloud data. This allows the generation unit 48 to generate an estimated plane with the smallest error for the extracted point cloud data. The generation unit 48 then outputs object information representing the position and orientation of the estimated plane.

[0093] The first portion of the designated object 100 may be a curved surface. For example, assume that the designated object 100 is a signboard attached to the side wall of a tunnel, and the first portion of the designated object 100 is the shape of a portion of the inner curved surface of a hollow cylinder having a central axis parallel to the lanes of a road. In this case, the generation unit 48 may constrain the central axis of the hollow cylinder expressed by the equation to be parallel to a coordinate axis parallel to the first portion in the reference coordinate system, and calculate an estimated curved surface that minimizes the distance to each of the multiple three-dimensional points included in the extracted point cloud data.

[0094] FIG. 8 is a flowchart showing the flow of processing by the generation unit 48 according to the fourth embodiment.

[0095] When it is unclear to which of the three coordinate axes the first portion of the designated object 100 is parallel, the generating unit 48 executes processing, for example, according to the flow shown in FIG.

[0096] First, in S31, the generation unit 48 acquires extracted point cloud data, a reference coordinate system, a target model, and regulating information. In this case, the regulating information indicates that a plane represented by the target model is parallel to one of the coordinate axes.

[0097] Next, the generation unit 48 executes the loop process between S32 and S35. In the loop process, the generation unit 48 executes the processes of S33 and S34 for each of the three coordinate axes (x-axis, y-axis, and z-axis).

[0098] In S33, the generation unit 48 constrains the plane represented by the equation to be parallel to the coordinate axes of the target, and estimates an estimated plane that minimizes the distance to each of the multiple 3D points included in the extracted point cloud data using the least squares method, etc. Subsequently, in S34, the generation unit 48 calculates the distance between the extracted point cloud data and the estimated plane.

[0099] When the generation unit 48 completes the processes of S33 and S34 for all three coordinate axes, it exits the loop process and advances the process to S36.

[0100] In S36, the generation unit 48 selects the estimated plane having the smallest distance from the extracted point cloud data from among the three estimated planes estimated with respect to the three coordinate axes. Then, in S37, the generation unit 48 outputs object information representing the position and orientation of the selected estimated plane.

[0101] As described above, when it is unclear to which of the three coordinate axes the first portion of the specified object 100 is parallel, the generation unit 48 generates an estimated plane for each of the three coordinate axes. Then, the generation unit 48 selects, from the three estimated planes generated for the three coordinate axes, the estimated plane that has the smallest distance from the extracted point cloud data, and outputs object information that represents the position and orientation of the selected estimated plane. In this way, the generation unit 48 can fit an object model to the first portion of the specified object 100, even when it is unclear to which of the three coordinate axes the first portion of the specified object 100 is parallel.

[0102] Even when the first portion of the specified object 100 is a curved surface, the generation unit 48 can estimate an estimated curved surface according to the flow shown in the flowchart of FIG. 8. For example, assume that the specified object 100 is a signboard attached to the side wall of a tunnel, and the first portion of the specified object 100 is the shape of a portion of the inner curved surface of a hollow cylinder having a central axis parallel to the lanes of a road. In this case, the generation unit 48 generates an estimated curved surface for each of three coordinate axes. Then, the generation unit 48 selects, from the three estimated curved surfaces generated for the three coordinate axes, the estimated curved surface that has the smallest distance from the extracted point cloud data, and outputs object information that represents the position and orientation of the selected estimated curved surface.

[0103] FIG. 9 is a diagram showing an example of a method for calculating an estimated plane when the first portion of the specified object 100, which is a plane, is perpendicular to the optical axis of the camera 210, a road dividing line, or the moving direction of the camera 210.

[0104] When a first portion of the designated object 100, which is a plane, is perpendicular to the optical axis of the camera 210, a road dividing line, or the movement direction of the camera 210, the first portion of the designated object 100 becomes parallel to two coordinate axes other than the coordinate axis parallel to the optical axis of the camera 210, the road dividing line, or the movement direction of the camera 210. In other words, when the direction parallel to the optical axis of the camera 210, the road dividing line, or the movement direction of the camera 210 is defined as the z-axis, the first portion of the designated object 100 becomes parallel to the x-axis and y-axis.

[0105] In such a case, the equation representing the plane has a coefficient of 0 multiplied by each of the variable (x) representing the distance from the origin in the direction of the x-axis and the variable (y) representing the distance from the origin in the direction of the y-axis. Therefore, in such a case, the equation representing the plane is expressed as z=C1, where C1 is a constant.

[0106] Therefore, in such a case, the generation unit 48 calculates the median of the positions in the z direction of the multiple three-dimensional points included in the extracted point cloud data, and calculates an estimated plane using the calculated median as a constant (C1). This allows the generation unit 48 to calculate the estimated plane through a very simple calculation. Note that the generation unit 48 may calculate an estimated plane using the average value as the constant (C1) instead of the median.

[0107] FIG. 10 is a diagram showing an example of a method for calculating an estimated plane when a first part of the specified object 100, which is a plane, is parallel to the optical axis of the camera 210, a road dividing line, or the direction of movement of the camera 210, and is perpendicular to the road.

[0108] When a first portion of the designated object 100, which is a plane, is parallel to the optical axis of the camera 210, a road dividing line, or the moving direction of the camera 210, and is perpendicular to the road, the first portion of the designated object 100 is parallel to a coordinate axis that is parallel to the optical axis of the camera 210, a road dividing line, or the moving direction of the camera 210, and is parallel to a coordinate axis that is perpendicular to the road. In other words, when a coordinate axis that is parallel to the optical axis of the camera 210, a road dividing line, or the moving direction of the camera 210 is defined as the z-axis, and a coordinate axis that is parallel to the direction perpendicular to the road is defined as the y-axis, the first portion of the designated object 100 is parallel to the z-axis and the y-axis.

[0109] In such a case, the equation representing the plane has a coefficient of 0 multiplied by each of the variable (z) representing the distance from the origin in the direction of the z-axis and the variable (y) representing the distance from the origin in the direction of the y-axis. Therefore, in such a case, the equation representing the plane is expressed as x = C2, where C2 is a constant.

[0110] Therefore, in such a case, the generation unit 48 calculates the median of the positions in the x direction of multiple three-dimensional points included in the extracted point cloud data, and calculates an estimated plane using the calculated median as a constant (C2). This allows the generation unit 48 to calculate the estimated plane through a very simple calculation. Note that the generation unit 48 may calculate an estimated plane using the average value as the constant (C2) instead of the median.

[0111] By calculating the estimated plane as described above, the generation unit 48 can accurately calculate the estimated plane even if the extracted point cloud data contains errors or if the extracted point cloud data contains three-dimensional points of other objects due to occlusion. For example, the generation unit 48 can accurately calculate the estimated plane even when three-dimensional point cloud data is generated based on image data.

[0112] For example, even if there is a tree plantation in front of the designated object 100 shown in Figure 9 or a pedestrian in front of the designated object 100 shown in Figure 10, the generation unit 48 can accurately calculate an estimated plane as long as the extracted point cloud data contains at least a majority of the three-dimensional points of the designated object 100.

[0113] (Fifth embodiment) Next, an information processing device 20 according to a fifth embodiment will be described.

[0114] 11 is a diagram showing a transformation vector from a model coordinate system to a reference coordinate system. In the fifth embodiment, a first portion of the specified object 100 has a three-dimensional shape. In the fifth embodiment, the object model is a three-dimensional model that represents the three-dimensional shape of the first portion, expressed in a model coordinate system different from the reference coordinate system.

[0115] A generation unit 48 according to the fifth embodiment calculates a transformation vector that minimizes the distance between each of the multiple 3D points included in the extracted point cloud data and the object model under the condition that the orientation of the object model matches the orientation defined for the first portion of the specified object 100. The transformation vector is a vector that performs coordinate transformation from the model coordinate system to the reference coordinate system.

[0116] 12 is a diagram showing an object model represented by a three-dimensional model. In the fifth embodiment, the designated object 100 is an object having a defined three-dimensional shape. In the fifth embodiment, the orientation of the designated object 100 with respect to at least one coordinate axis in a reference coordinate system is defined.

[0117] For example, in the fifth embodiment, the designated object 100 is a traffic light that allows or stops the travel of the vehicle 200. The traffic light is installed so that the light emission direction of the multiple signal lights is parallel to the travel direction of the vehicle 200. Therefore, the designated object 100 that is a traffic light is defined as the light emission direction of the multiple signal lights being parallel to the optical axis direction of the camera 210, the road dividing line, or a coordinate axis that is parallel to the travel direction of the camera 210.

[0118] Here, the specified object 100 has multiple signal lights whose light emission directions are parallel to the z-axis in the reference coordinate system. In this case, the transformation matrix for transforming coordinates from the model coordinate system to the reference coordinate system can have a rotation angle about the y-axis of 0. Therefore, the generation unit 48 according to the fifth embodiment can fit the object model to the extracted point cloud data by constraining some components of the transformation matrix to predetermined values. This allows the generation unit 48 according to the fifth embodiment to easily and accurately fit the object model, which is a three-dimensional model.

[0119] (Sixth embodiment) Next, an information processing device 20 according to a sixth embodiment will be described.

[0120] 13 is a diagram showing the functional configuration of an information processing device 20 according to a sixth embodiment. In the sixth embodiment, the orientation of the designated object 100 is defined with respect to a reference coordinate system. In the sixth embodiment, the information processing device 20 further includes an orientation detection unit 64 in addition to the configuration of the information processing device 20 according to the first embodiment.

[0121] The orientation detection unit 64 detects the orientation of the first portion of the designated object 100. The orientation detection unit 64 detects the orientation of the designated object 100 using an object detection technique based on image data. Alternatively, the orientation detection unit 64 may detect the orientation of the designated object 100 based on information input by a user.

[0122] In the sixth embodiment, the generation unit 48 further acquires the orientation detected by the orientation detection unit 64. In the sixth embodiment, the generation unit 48 generates object information by fitting the object model to the extracted point cloud data under the condition that the orientation of the object model matches the orientation of the detected first portion of the specified object 100. This allows the generation unit 48 to calculate the distance between the extracted point cloud data and the object model in a single process, without having to calculate the distance between the extracted point cloud data and the object model for all three coordinate axes as in the flowchart shown in FIG. 8, for example. This allows the information processing device 20 according to the sixth embodiment to generate object information with high accuracy through simple processing.

[0123] Seventh embodiment Next, an information processing device 20 according to the seventh embodiment will be described.

[0124] 14 is a diagram showing the functional configuration of an information processing device 20 according to the seventh embodiment. In the seventh embodiment, the information processing device 20 further includes a size calculation unit 68 in addition to the configuration of the information processing device 20 according to the first embodiment.

[0125] The size calculation unit 68 acquires the target area detected by the area detection unit 40 and the object information generated by the generation unit 48. The size calculation unit 68 calculates the size of the designated object 100 based on the target area and the object information. The size of the designated object 100 is, for example, one or more values ​​of the height (H) of the designated object 100, the width (W) of the designated object 100, or the depth of the designated object 100.

[0126] The output unit 50 acquires the size from the size calculation unit 68. The output unit 50 outputs the size together with the object information.

[0127] FIG. 15 is a diagram showing a size calculation method.

[0128] For example, when the object model is information representing a plane, the size calculation unit 68 projects the object region in the image plane represented by the image data onto the estimated plane. Then, the size calculation unit 68 calculates the size by measuring the height (H) and width (W) of the object region projected onto the estimated plane. Even when the object model is a curved surface, the size calculation unit 68 calculates the size by measuring the distance or Euclidean distance of the height (H) and width (W) on the curved surface of the object region projected onto the estimated curved surface.

[0129] According to the information processing device 20 of the seventh embodiment, the control device 230 can recognize the information indicated on the designated object 100 based on the object information and size, and control the vehicle 200, or display information about the designated object 100 to the driver, etc.

[0130] (Hardware configuration of information processing device) FIG. 16 is a diagram illustrating an example of the hardware configuration of the information processing device 20 according to each embodiment.

[0131] The information processing device 20 is realized by a computer having a hardware configuration such as that shown in Fig. 16. The information processing device 20 includes a CPU (Central Processing Unit) 301, a RAM (Random Access Memory) 302, a ROM (Read Only Memory) 303, an operation input device 304, a display device 305, a storage device 306, and a communication device 307. These components are connected via a bus.

[0132] The CPU 301 is a processor that executes arithmetic processing, control processing, etc. in accordance with a program. The CPU 301 uses a predetermined area of ​​the RAM 302 as a work area and executes various processes in cooperation with programs stored in the ROM 303, the storage device 306, etc.

[0133] The RAM 302 is a memory such as an SDRAM (Synchronous Dynamic Random Access Memory), and functions as a work area for the CPU 301. The ROM 303 is a memory that stores programs and various types of information in a non-rewritable manner.

[0134] The operation input device 304 is an input device such as a mouse, a keyboard, etc. The operation input device 304 receives information input by a user as an instruction signal, and outputs the instruction signal to the CPU 301.

[0135] The display device 305 is a display device such as an LCD (Liquid Crystal Display), etc. The display device 305 displays various information based on a display signal from the CPU 301.

[0136] The storage device 306 is a device that writes and reads data to a semiconductor storage medium such as a flash memory, or a magnetically or optically recordable storage medium, etc. The storage device 306 writes and reads data to the storage medium in response to control from the CPU 301. The communication device 307 communicates with external devices via a network in response to control from the CPU 301.

[0137] The program executed by the computer has a modular configuration including an image acquisition module, a point cloud acquisition module, a coordinate system identification module, an object setting module, a region detection module, an extraction module, a model acquisition module, a generation module, and an output module. The program may further include a motion detection module, a posture detection module, and a size calculation module.

[0138] This program, when deployed on RAM 302 and executed by CPU 301 (processor), causes the computer to function as image acquisition unit 32, point cloud acquisition unit 34, coordinate system specification unit 36, object setting unit 38, area detection unit 40, extraction unit 42, model acquisition unit 46, generation unit 48, and output unit 50. Furthermore, this program may cause the computer to function as motion detection unit 60, posture detection unit 64, and size calculation unit 68. Note that some or all of image acquisition unit 32, point cloud acquisition unit 34, coordinate system specification unit 36, object setting unit 38, area detection unit 40, extraction unit 42, model acquisition unit 46, generation unit 48, and output unit 50 may be realized by hardware circuits.

[0139] In addition, the program to be executed by a computer is provided as a file in a format that can be installed on a computer or in a format that can be executed by a computer, and is recorded on a computer-readable recording medium such as a CD-ROM, a flexible disk, a CD-R, or a DVD (Digital Versatile Disk).

[0140] This program may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. This program may also be provided or distributed via a network such as the Internet. The program executed by the information processing device 20 may also be provided by being pre-installed in the ROM 303 or the like.

[0141] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as defined in the claims.

[0142] (Addendum) The above-described embodiments can be summarized as the following technical proposals.

[0143] [Technical proposal 1] A program for causing a computer to function as an information processing device, The information processing device an image acquisition unit that acquires image data; a point cloud acquisition unit that acquires three-dimensional point cloud data including a plurality of three-dimensional points each representing a three-dimensional position of an object included in the image data; a coordinate system specifying unit that specifies a reference coordinate system that represents a reference for a three-dimensional position; an area detection unit that detects a two-dimensional object area including a specified object in the image data; an extraction unit that extracts extracted point cloud data representing three-dimensional positions of objects included in the target area from the three-dimensional point cloud data; a model acquisition unit that acquires an object model that is information that models the shape of a first portion that is at least a part of the specified object; a generation unit that generates object information representing a position and an orientation of the object model when the object model is placed in a three-dimensional space in correspondence with the specified object; an output unit that outputs the object information; and make it work, the specified object has a first portion whose shape is defined and a posture of the first portion defined with respect to at least one coordinate axis in the reference coordinate system; The generation unit generates the object information by fitting the object model to the extracted point cloud data under a condition that the orientation of the object model matches an orientation defined for the first portion of the specified object. program.

[0144] [Technical proposal 2] the image data is generated by a camera mounted on a moving object; The reference coordinate system has three coordinate axes, one of which is parallel to the optical axis of the camera. The program described in Technical Proposal 1.

[0145] [Technical proposal 3] the image data is generated by a camera mounted on a mobile object traveling on a road, One of the three coordinate axes of the reference coordinate system is parallel to the dividing line of the road. The program described in Technical Proposal 1.

[0146] [Technical proposal 4] the image data is generated by a camera mounted on a moving object; The reference coordinate system has three coordinate axes, one of which is parallel to the direction of movement of the camera. The program described in Technical Proposal 1.

[0147] [Technical proposal 5] The direction of movement is calculated based on the movement of the camera. The program described in Technical Proposal 4.

[0148] [Technical proposal 6] The first portion of the specified object is a plane that is parallel to one or two of the three coordinate axes in the reference coordinate system. A program according to any one of technical proposals 1 to 5.

[0149] [Technical proposal 7] The object model is the equation of the plane The program described in Technical Proposal 6.

[0150] [Technical proposal 8] The generation unit constrains the plane represented by the equation to be parallel to a coordinate axis parallel to the first portion in the reference coordinate system, generates an estimated plane that minimizes a distance to each of the plurality of three-dimensional points included in the extracted point cloud data, and generates the object information that indicates a position and orientation of the generated estimated plane. The program described in Technical Proposal 7.

[0151] [Technical proposal 9] The first portion of the specified object has a shape of a part of the inner curved surface of a hollow cylinder, and the hollow cylinder forming the first portion has a central axis parallel to one of three coordinate axes in the reference coordinate system. A program according to any one of technical proposals 1 to 5.

[0152] [Technical proposal 10] The object model is an equation of the inner curved surface of the hollow cylinder. The program described in Technical Proposal 9.

[0153] [Technical proposal 11] The generation unit generates an estimated curved surface that minimizes the distance to each of the plurality of three-dimensional points included in the extracted point cloud data by constraining the central axis of the hollow cylinder represented by the equation to be parallel to a coordinate axis that is parallel to the central axis of the hollow cylinder that forms the first portion, and generates the object information that indicates the position and orientation of the generated estimated curved surface. The program described in Technical Proposal 10.

[0154] [Technical proposal 12] The object model represents a three-dimensional shape of the first portion of the specified object. A program according to any one of technical proposals 1 to 5.

[0155] [Technical proposal 13] the object model is represented by a model coordinate system different from the reference coordinate system; The generation unit calculates a transformation vector for coordinate transformation from the model coordinate system to the reference coordinate system that minimizes the distance between each of a plurality of three-dimensional points included in the extracted point cloud data and the object model under a condition that the orientation of the object model matches an orientation defined for the first portion of the specified object. The program described in Technical Proposal 12.

[0156] [Technical proposal 14] The information processing device includes: further comprising a posture detection unit that detects a posture of the first portion of the designated object; The generation unit generates the object information by fitting the object model to the extracted point cloud data under a condition that the orientation of the object model matches the orientation of the first part of the detected specified object. A program according to any one of technical proposals 1 to 13.

[0157] [Technical proposal 15] The information processing device includes: a size calculation unit that calculates a size of the specified object in the object model based on the object region and the object information; The output unit outputs the size together with the object information. A program according to any one of technical proposals 1 to 14.

[0158] [Technical proposal 16] The point cloud acquisition unit generates the three-dimensional point cloud data based on the image data. A program according to any one of technical proposals 1 to 15.

[0159] [Technical proposal 17] The point cloud acquisition unit acquires the three-dimensional point cloud data, in which a correspondence relationship with pixel positions in the image data is determined, from a three-dimensional sensor device. A program according to any one of technical proposals 1 to 15.

[0160] [Technical proposal 18] an image acquisition unit that acquires image data; a point cloud acquisition unit that acquires three-dimensional point cloud data including a plurality of three-dimensional points each representing a three-dimensional position of an object included in the image data; a coordinate system specifying unit that specifies a reference coordinate system that represents a reference for a three-dimensional position; an area detection unit that detects a two-dimensional object area including a specified object in the image data; an extraction unit that extracts extracted point cloud data representing three-dimensional positions of objects included in the target area from the three-dimensional point cloud data; a model acquisition unit that acquires an object model that is information that models the shape of a first portion that is at least a part of the specified object; a generation unit that generates object information representing a position and an orientation of the object model when the object model is placed in a three-dimensional space in correspondence with the specified object; an output unit that outputs the object information; Equipped with the specified object has a first portion whose shape is defined and a posture of the first portion defined with respect to at least one coordinate axis in the reference coordinate system; The generation unit generates the object information by fitting the object model to the extracted point cloud data under a condition that the orientation of the object model matches an orientation defined for the first portion of the specified object. Information processing device.

[0161] [Technical proposal 19] An information processing method executed by an information processing device, The information processing device acquires image data, the information processing device acquires three-dimensional point cloud data including a plurality of three-dimensional points each representing a three-dimensional position of an object included in the image data; the information processing device specifies a reference coordinate system that represents a reference for a three-dimensional position; the information processing device detects a two-dimensional target area in the image data that includes a specified target; the information processing device extracts, from the three-dimensional point cloud data, extracted point cloud data that represents three-dimensional positions of objects included in the target area; the information processing device acquires an object model, which is information that models the shape of a first portion that is at least a part of the specified object; the information processing device generates object information representing a position and an orientation of the object model when the object model is placed in a three-dimensional space in correspondence with the specified object; the information processing device outputs the object information; Equipped with the specified object has a first portion whose shape is defined and a posture of the first portion defined with respect to at least one coordinate axis in the reference coordinate system; In generating the object information, the information processing device generates the object information by fitting the object model to the extracted point cloud data under a condition that the orientation of the object model coincides with an orientation defined for the first portion of the specified object. Information processing methods. [Explanation of symbols]

[0162] 20 Information processing equipment 32 Image acquisition unit 34 Point cloud acquisition section 36 Coordinate system identification section 38 Object setting section 40 Area detection unit 42 Extraction part 44 Model memory section 46 Model Acquisition Department 48 Generation part 50 Output section 60 Motion detection unit 64 Attitude detection unit 68 Size Calculation Section 100 Designated Objects 200 vehicles 210 Camera 220 Three-dimensional sensor device 230 Control device

Claims

1. A program for causing a computer to function as an information processing device, The information processing device an image acquisition unit that acquires image data; a point cloud acquisition unit that acquires three-dimensional point cloud data including a plurality of three-dimensional points each representing a three-dimensional position of an object included in the image data; a coordinate system specifying unit that specifies a reference coordinate system that represents a reference for a three-dimensional position; an area detection unit that detects a two-dimensional object area including a specified object in the image data; an extraction unit that extracts extracted point cloud data representing three-dimensional positions of objects included in the target area from the three-dimensional point cloud data; a model acquisition unit that acquires an object model, which is information that models the shape of a first portion that is at least a part of the specified object; a generation unit that generates object information representing a position and an orientation of the object model when the object model is placed in a three-dimensional space in correspondence with the specified object; an output unit that outputs the object information; and make it work, the specified object has a first portion whose shape is defined and a posture of the first portion defined with respect to at least one coordinate axis in the reference coordinate system; The generation unit generates the object information by fitting the object model to the extracted point cloud data under a condition that the orientation of the object model coincides with an orientation defined for the first portion of the specified object. program.

2. the image data is generated by a camera mounted on a moving object; The reference coordinate system has three coordinate axes, one of which is parallel to the optical axis of the camera. The program according to claim 1.

3. the image data is generated by a camera mounted on a mobile object traveling on a road, One of the three coordinate axes of the reference coordinate system is parallel to the dividing line of the road. The program according to claim 1.

4. the image data is generated by a camera mounted on a moving object; The reference coordinate system has three coordinate axes, one of which is parallel to the direction of movement of the camera. The program according to claim 1.

5. The direction of movement is calculated based on the movement of the camera. The program according to claim 4.

6. The first portion of the specified object is a plane that is parallel to one or two of the three coordinate axes in the reference coordinate system. The program according to claim 1.

7. The object model is the equation of the plane The program according to claim 6.

8. The generation unit constrains the plane represented by the equation to be parallel to a coordinate axis parallel to the first portion in the reference coordinate system, to generate an estimated plane that minimizes a distance to each of the plurality of three-dimensional points included in the extracted point cloud data, and generates the object information that indicates a position and orientation of the generated estimated plane. The program according to claim 7.

9. The first portion of the specified object has a shape of a part of an inner curved surface of a hollow cylinder, and the hollow cylinder forming the first portion has a central axis parallel to one of three coordinate axes in the reference coordinate system. The program according to claim 1.

10. The object model is an equation of the inner curved surface of the hollow cylinder. The program according to claim 9.

11. The generation unit constrains the central axis of the hollow cylinder represented by the equation to be parallel to a coordinate axis that is parallel to the central axis of the hollow cylinder that forms the first portion, and generates an estimated curved surface that minimizes the distance to each of the plurality of three-dimensional points included in the extracted point cloud data, and generates the object information that represents the position and orientation of the generated estimated curved surface. The program according to claim 10.

12. The object model represents a three-dimensional shape of the first portion of the specified object. The program according to claim 1.

13. the object model is represented by a model coordinate system different from the reference coordinate system; The generation unit calculates a transformation vector for transforming coordinates from the model coordinate system to the reference coordinate system, which minimizes the distance between each of a plurality of three-dimensional points included in the extracted point cloud data and the object model, under the condition that the orientation of the object model coincides with the orientation defined for the first portion of the specified object. The program according to claim 12.

14. The information processing device includes: further comprising a posture detection unit that detects a posture of the first portion of the designated object; The generation unit generates the object information by fitting the object model to the extracted point cloud data under a condition that the orientation of the object model matches the orientation of the first part of the detected specified object. The program according to claim 1.

15. The information processing device includes: a size calculation unit that calculates a size of the specified object in the object model based on the object region and the object information; The output unit outputs the size together with the object information. The program according to claim 1.

16. The point cloud acquisition unit generates the three-dimensional point cloud data based on the image data. The program according to claim 1.

17. The point cloud acquisition unit acquires the three-dimensional point cloud data, in which a correspondence relationship with pixel positions in the image data is determined, from a three-dimensional sensor device. The program according to claim 1.

18. an image acquisition unit that acquires image data; a point cloud acquisition unit that acquires three-dimensional point cloud data including a plurality of three-dimensional points each representing a three-dimensional position of an object included in the image data; a coordinate system specifying unit that specifies a reference coordinate system that represents a reference for a three-dimensional position; an area detection unit that detects a two-dimensional object area including a specified object in the image data; an extraction unit that extracts extracted point cloud data representing three-dimensional positions of objects included in the target area from the three-dimensional point cloud data; a model acquisition unit that acquires an object model, which is information that models the shape of a first portion that is at least a part of the specified object; a generation unit that generates object information representing a position and an orientation of the object model when the object model is placed in a three-dimensional space in correspondence with the specified object; an output unit that outputs the object information; Equipped with the specified object has a first portion whose shape is defined and a posture of the first portion defined with respect to at least one coordinate axis in the reference coordinate system; The generation unit generates the object information by fitting the object model to the extracted point cloud data under a condition that the orientation of the object model coincides with an orientation defined for the first portion of the specified object. Information processing device.

19. An information processing method executed by an information processing device, The information processing device acquires image data, the information processing device acquires three-dimensional point cloud data including a plurality of three-dimensional points each representing a three-dimensional position of an object included in the image data; the information processing device specifies a reference coordinate system that represents a reference for a three-dimensional position; the information processing device detects a two-dimensional target area in the image data that includes a specified target; the information processing device extracts, from the three-dimensional point cloud data, extracted point cloud data that represents three-dimensional positions of objects included in the target area; the information processing device acquires an object model, which is information that models a shape of a first portion that is at least a part of the specified object; the information processing device generates object information representing a position and an orientation of the object model when the object model is placed in a three-dimensional space in correspondence with the specified object; the information processing device outputs the object information; the specified object has a first portion whose shape is defined and a posture of the first portion defined with respect to at least one coordinate axis in the reference coordinate system; In generating the object information, the information processing device generates the object information by fitting the object model to the extracted point cloud data under a condition that the orientation of the object model coincides with an orientation defined for the first portion of the specified object. Information processing methods.

Images