Information processing method, information processing device, and program

Abstract

This provides a technique advantageous for assigning attribute information to 3D models. In an information processing method for assigning attribute information to a 3D model of an object, the area and normal direction of each face of the 3D model are calculated, the viewing direction is determined based on the calculated area size and normal direction, an image of the 3D model viewed from the viewing direction is acquired, and the image viewed from the viewing direction is input to a trained model that outputs attribute information from input images to obtain attribute information of the 3D model and assign attribute information to the 3D model.

Description

Technical Field

[0001] The present invention relates to an information processing method, an information processing apparatus, and a program for attaching attribute information to a 3D model.

Background Art

[0002] Conventionally, software for inputting attribute information such as windows and doors into a 3D model such as BIM (Building Information Modeling) by a user's operation has been used.

[0003] Patent Document 1 discloses generating a 3D point cloud using a plurality of images taken from above, and generating a texture map representing the 3D shape of a roof based on the generated 3D point cloud. Further, Patent Document 1 discloses classifying roof features existing in an image using artificial intelligence, performing a coloring process such that individual different roof structure features are uniquely colored, applying a coloring classification label corresponding to the roof structure feature to the point cloud, and indicating a specific roof structure feature with a specific color.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] Inputting attribute information into a large number of 3D models by a user's operation in software is time-consuming and costly, and input errors may occur. In addition, although there is a method of classifying roofs using artificial intelligence as described in Patent Document 1, it is difficult to extract detailed attribute information other than roofs with higher accuracy for a wide variety of 3D models, and there is a possibility of incorrect output.

[0006] Therefore, the present invention aims to provide a technique that is advantageous for assigning attribute information to a three-dimensional model. [Means for solving the problem]

[0007] An information processing method, as one aspect of the present invention for solving the above problems, is an information processing method for assigning attribute information to a 3D model of an object, characterized in that a computer calculates the area and normal direction of each face of the 3D model, determines the viewing direction based on the calculated area and normal direction, acquires an image of the 3D model viewed from the viewing direction, inputs the image viewed from the viewing direction to a trained model that outputs attribute information from an input image, obtains attribute information of the 3D model, and assigns the attribute information to the 3D model. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide a technique that is advantageous for assigning attribute information to a three-dimensional model. [Brief explanation of the drawing]

[0009] [Figure 1] This is a diagram showing the configuration of an information processing device. [Figure 2] This is a diagram illustrating a flowchart of an information processing method. [Figure 3] This is a diagram representing the TIN model. [Figure 4] This is a diagram representing the normal vector. [Figure 5] This diagram shows the range of angles for each normal vector. [Figure 6] This is a diagram illustrating the concept of weight peaks. [Figure 7] This is a diagram representing a three-dimensional model of a building. [Figure 8] This diagram illustrates the bounding box, line of sight, and viewpoint. [Figure 9] This diagram represents a 2D image of a 3D model as seen from the direction of the line of sight. [Figure 10]It is a diagram showing a two-dimensional image of a three-dimensional model viewed from a second line-of-sight direction. [Figure 11] It is a diagram showing an image of a three-dimensional model viewed from the upper surface direction of a bounding box in a comparative example. [Figure 12] It is a diagram showing an image of a three-dimensional model viewed from the lower surface direction of a bounding box in a comparative example.

Embodiments for Carrying Out the Invention

[0010] Embodiments of the present invention include, for example, the following configurations.

[0011] [Item 1] In an information processing method for assigning attribute information to a three-dimensional model of an object, a computer calculates the area and normal direction of each surface of the three-dimensional model, determines the line-of-sight direction based on the calculated area size and the normal direction, acquires an image of the three-dimensional model viewed from the line-of-sight direction, inputs the image viewed from the line-of-sight direction into a learned model that outputs attribute information from an input image, obtains the attribute information of the three-dimensional model, and assigns the attribute information to the three-dimensional model. An information processing method characterized by this. [Item 2] For each surface of the three-dimensional model, based on the direction of a first component vector that is a normal vector corresponding to the surface with the largest total area among surfaces facing the same range of directions, the information processing method according to Item 1, characterized in that the line-of-sight direction is determined. [Item 3] The information processing method according to Item 1, characterized in that the direction along the first component vector starting from the center of the three-dimensional model is determined as the line-of-sight direction. [Item 4] Determine a plurality of the line-of-sight directions, input a plurality of images viewed from the plurality of the line-of-sight directions into the learned model, and obtain the attribute information corresponding to the plurality of images. The information processing method according to Item 1, characterized by this. [Item 5] The information processing method according to claim 1, characterized in that the attribute information is obtained from an image of the three-dimensional model including the geospatial information, which is overlaid with the three-dimensional model of the object and viewed from the line-of-sight direction. [Item 6] A program for causing a computer to execute the information processing method according to any one of Items 1 to 5. [Item 7] In an information processing apparatus for assigning attribute information to a three-dimensional model of an object, it has a processing unit, and the processing unit calculates the area and normal direction of each face of the three-dimensional model, determines the line-of-sight direction based on the calculated area size and the normal direction, acquires an image of the three-dimensional model viewed from the line-of-sight direction, inputs the image viewed from the line-of-sight direction into a learned model that outputs attribute information from an input image, obtains the attribute information of the three-dimensional model, and assigns the attribute information to the three-dimensional model.

[0012] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the present specification and drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant description is omitted.

[0013] <Details of Embodiment> This embodiment describes a technique for assigning attribute information to objects represented by three-dimensional data, such as a three-dimensional (3D) model or a three-dimensional point cloud. Examples of objects include, but are not limited to, buildings such as detached houses, apartments, condominiums, skyscrapers, shops, commercial facilities, and factories; structures such as roads, transportation facilities, bridges, dams, wind power generation facilities, power transmission towers, communication towers, and agricultural facilities; and natural objects such as topography, forests, and farmland (fields, rice paddies, and orchards). Attribute information, unlike the shape or location of the object itself, provides descriptive data and characteristics about the object and is used to understand, identify, classify, and analyze the object in more detail. It can include various properties and states that are directly or indirectly related to the object. Examples of adding attribute information to 3D data include, but are not limited to, the following. Building data can be assigned attribute information to each component, such as walls, windows, doors, entrances, balconies, roofs, chimneys, antennas, air conditioning equipment, decks, columns, beams, and solar panels. This information includes material type (reinforced concrete, wood, etc.), strength, height, year of installation, and number of floors. It can also include gardens, swimming pools, barns, and patios associated with the building. This information is then used for structural analysis, maintenance planning, and urban planning. Geological attributes (type of soil layer, soil strength, groundwater location, etc.) can be assigned to topographic data. This information is then used for civil engineering works and disaster risk assessment. Forest data can be assigned information such as plant and tree species, height, and health status. This facilitates forest management and monitoring of vegetation health. As these examples show, adding attribute information gives 3D data more meaning and makes it more useful.

[0014] Figure 1 shows a configuration diagram of the information processing device 1 in this embodiment. The information processing device 1 has a computer that can be installed in a general-purpose computer such as a personal computer, or in a device such as a smartphone or tablet. The information processing device 1 executes an information processing method that assigns attribute information to an object represented as 3D data using installed software or applications (programs).

[0015] The information processing device 1 includes a processing unit 11, a memory 12, a storage unit 13, a communication unit 14, an input unit 15, and a display unit 16. These are electrically connected to each other via a bus 17.

[0016] The processing unit 11 is a computing unit that controls the operation of the entire information processing device 1, controls the transmission and reception of data between each part, and performs information processing necessary for program execution and authentication processing. The processing unit 11 includes a computing unit that is, for example, a processor such as a CPU, GPU, or FPGA, and executes programs stored in the storage 13 and loaded into the memory 12 to perform various information processing described later.

[0017] The memory 12 (storage unit) includes a main memory composed of a volatile storage device such as DRAM, and an auxiliary memory composed of a non-volatile storage device such as flash memory or an HDD. The memory 12 is used as a work area for the processing unit 11, and also stores the BIOS and various setting information that are executed when the information processing device 1 is started up.

[0018] Storage 13 (storage medium) includes storage devices such as HDDs and SSDs, and stores various programs such as applications and programs. In addition, a database containing data used for each process is built on storage 13.

[0019] The communication unit 14 connects the information processing device 1 to a network. The communication unit 14 communicates with external devices directly or via a network access point using methods such as wired LAN, wireless LAN, Wi-Fi (Wireless Fidelity, registered trademark), infrared communication, Bluetooth (registered trademark), short-range or contactless communication. The input unit 15 is an information input device such as a keyboard, mouse, or touch panel.

[0020] The display unit (display device) 16 includes a display that shows information calculated by the processing unit 11 or information received from an external source by the communication unit 14. The display unit 16 provides a graphical user interface (GUI) that displays various information on its screen. The display unit 16 is not limited to being integrated with the information processing device 1, but may also be a display (display device) provided separately from the information processing device 1 and connected to the information processing device 1.

[0021] Bus 17 is connected in common to all of the above components and transmits, for example, address signals, data signals, and various control signals.

[0022] Next, we will explain an information processing method for assigning attribute information to an object represented by 3D data. Figure 2 shows a flowchart of this method.

[0023] First, the information processing device 1 acquires 3D data for assigning attribute information and creates a 3D model (S1). The 3D data includes 3D point cloud data where each point cloud represents a 3D coordinate. Methods for creating this 3D data include laser scanning methods such as LiDAR, methods for calculating 3D data from multiple images or videos taken from multiple different angles such as Structure from Motion (SfM), methods for projecting patterned light (such as stripes) onto an object and capturing the distortion when the light reflects off the object with a camera to determine the 3D shape, and methods for calculating distance by shining light onto an object and measuring the time it takes for the light to reflect back.

[0024] From here on, we will explain the SfM method as an example. First, the target object is photographed using a camera (imaging device) on a mobile device such as a drone, and image data is acquired. The camera on the mobile device photographs the target object from various angles. The photography can be done by shooting video using the camera on the mobile device, or by taking multiple still images. Also, if a certain distance is maintained between the target object and the camera when taking photographs, the 3D information will be reconstructed more accurately.

[0025] Furthermore, filming may be performed by the mobile body moving autonomously, by the mobile body being moved automatically by a control device, or by a person operating a control device to control the position and orientation of the mobile body. When the mobile body is moved automatically, the control device sets waypoints that include points along the path the mobile body is moving, and controls the position of the mobile body so that it moves according to the waypoints. The control device transmits the waypoints of the mobile body as target positions to the mobile body and performs position control on the mobile body, for example, according to the difference between the target position and the measured current position. The control device can also control the shooting conditions of the mobile body's camera during video recording, such as the orientation (angle, direction), focal position, brightness, and sensitivity.

[0026] Image data captured by a mobile camera is transmitted to an information processing device (computer) that performs SfM. If necessary, pre-processing can be performed on the captured images before inputting the data into the SfM algorithm. For example, unwanted images may be removed, or the brightness and contrast of the images may be adjusted.

[0027] Next, the information processing device that performs SfM uses multiple captured or pre-processed images to perform SfM. First, it uses multiple images to detect feature points in the images. Feature points are unique points in an image that are easy to match from different viewpoints, such as the edges and corners of an object. Common algorithms include SIFT (Scale-Invariant Feature Transform) and SURF (Speeded Up Robust Features).

[0028] Next, the feature points detected from each image are mapped across multiple images. Common points in images taken from different viewpoints are matched to identify which parts of which images represent the same point. Feature-based methods (such as feature descriptors like SIFT or SURF) or nearest neighbor search algorithms are used for matching.

[0029] Then, from the positional information (2D coordinates) of feature points matched between the two images, the camera's position and orientation (angle, direction) at the time each image was taken is estimated. For example, a mathematical model called epipolar geometry is used to calculate the relative position and orientation of the cameras by integrating information from multiple viewpoints based on parallax. Basic matrices and essential matrices are calculated to estimate the relative position and orientation between cameras. Alternatively, the camera's position and orientation can be calculated using AI such as Gaussian Splatting. In this way, the camera's position and orientation in 3D space are determined, and the image and the camera's position and orientation at the time the image was taken can be associated and stored in a database or other storage unit.

[0030] Furthermore, a 3D point cloud is constructed using feature points matched from each viewpoint. Specifically, based on the estimated camera position and orientation, the coordinates of corresponding 3D points are calculated from the coordinates of matching 2D feature points across multiple images. This process is based on triangulation. This method allows us to determine where the features of an object seen in a 2D image are located in 3D space. The 3D point cloud is represented by 3D coordinates, with each point representing a position corresponding to the surface of the object. This allows us to grasp the approximate shape of the object. If necessary, by simultaneously adjusting the positions of all cameras and the 3D point cloud and performing bundle adjustment to minimize errors, more accurate camera positions and 3D point clouds can be obtained.

[0031] The 3D point cloud obtained at this stage is sparsely datated, so to obtain a denser point cloud, it is necessary to find and add more corresponding points. Algorithms such as stereo matching and optical flow can be used for this purpose. Stereo matching uses two or more image pairs to find corresponding points for each pixel and generate a dense point cloud. Optical flow calculates the movement of pixels between temporally consecutive images to generate a more detailed point cloud. It is also possible to obtain a dense 3D point cloud using methods such as Patch-based Multi-view Stereo (PMVS), which estimates the position of each point using the pixel information of the image. Furthermore, if the point cloud contains noise such as outliers, noise removal and optimization may be performed.

[0032] Since the camera position and 3D point cloud calculated by SfM are estimated relatively, to obtain the actual size and distance, one method is to use external scale information (e.g., dimensions of known objects, GPS data). For example, when photographing the object, known survey points or targets can be set up in advance at the shooting site, and after SfM is completed, their coordinate values ​​can be entered and fitted. Alternatively, targets such as AR markers that indicate arbitrary coordinates can be placed in the image data during SfM processing, and these coordinate values ​​can be provided during SfM processing for fitting, thereby representing the output of the SfM processing in a real coordinate system.

[0033] Next, the information processing device 1 generates a three-dimensional model (3D model) by connecting points based on the constructed point cloud data. Modeling is a step to represent the shape smoothly by polygonizing the 3D point cloud, so that the detailed shape of the object is more clearly represented. The 3D model is composed of, for example, a TIN (Triangular Irregular Network) model. A TIN model is a model composed of multiple triangular planar figures that connect three points that are close together from the obtained 3D point cloud, and these are connected to each other without intersecting, forming a digital data structure that represents 3D data as a collection of triangles. The surface shape of the TIN model represents the surface shape of the measured object. Alternatively, the 3D model may be a model represented by two-dimensional polygonal figures such as quadrilaterals, or a model represented by a mesh of vertices, edges, and faces (polygons).

[0034] The 3D point cloud and 3D model obtained as described above can be output in various formats.

[0035] Next, the processing unit of the information processing device 1 overlays the acquired 3D model or 3D point cloud data with an image representing 2D geographic information (S2). Geographic information can be obtained from a Geographic Information System (GIS), which handles data containing various location-related information, including map information and various statistical information, on an electronic map. The overlay in S2 is not necessarily required if geographic information is not available or if it is not necessary for attribute information assignment.

[0036] Next, the processing unit of the information processing device 1 performs surface analysis on the acquired 3D model (S3). The 3D model is represented, for example, as a TIN model. A TIN model is composed of a set of faces where many triangular planes are adjacent, and each triangle has a unique area and face orientation. Figure 3 shows a diagram representing a part of a TIN model. A TIN model is represented as a set of triangles t formed by connecting each point p in the 3D point cloud with a line (edge) e. In S3, first, the area of ​​each triangle constituting the model and the normal vector v perpendicular to the plane of each triangle are calculated using the model data. The area and normal vector can be calculated using the coordinates of the three vertices of the triangle. Here, the normal vector is represented as a vector with 3D components and is a unit vector with magnitude r = 1. Figure 4 shows a diagram of the coordinate representation of the normal vector. The normal vector can be represented in coordinates (X1, Y1, Z1) of a 3D Cartesian coordinate system, or in spherical coordinates with azimuth angle φ and elevation angle Θ. The direction of the normal vector can be set to point outwards from the 3D model.

[0037] The processing unit of the information processing device 1 assigns the magnitude of the area as weight information to the calculated normal vector of each triangle in the plane. For example, the normal vector of a certain triangle t1 and the area of ​​that triangle t1 are associated as a set of data, and the normal vector of another triangle t2 and the area of ​​that triangle t2 are associated as a set of data, and the data is saved.

[0038] Next, the weights of multiple normal vectors are statistically processed. In the statistical processing, the sum of the magnitudes of the weights of the normal vectors contained in each range, which is obtained by dividing the 3D space into ranges at regular intervals, is calculated. Each range can be set as a range by dividing the azimuth angle and elevation angle into ranges d at regular intervals, similar to dividing the surface of a sphere of radius 1 into ranges at regular angular intervals, as shown in Figure 5. For example, the azimuth angle can be divided into 360 equal parts to set ranges of 0 degrees to less than 1 degree, 1 degree to less than 2 degrees, ..., 359 degrees to less than 360 degrees, and the elevation angle can be similarly divided into 360 equal parts to set ranges where the azimuth angle and elevation angle are divided into 1-degree intervals. Alternatively, the angle can be rounded to the first decimal place to obtain an integer angle. That is, the range from 0 to less than 0.5 degrees can be set to 0, and the range from 0.5 to less than 1.5 degrees can be set to 1. Although the angle is expressed in degrees, it can also be expressed in radians. Furthermore, each range can be defined as a fixed interval in the X, Y, and Z coordinates of a three-dimensional Cartesian coordinate system, covering the surface of a sphere with radius 1. The sum of the weights in each range represents the sum of the area of ​​the triangular faces with normal vectors contained within that range.

[0039] The processing unit of the information processing device 1 can calculate statistical values ​​such as the range in which the sum of the weights has a maximum value, the range in which the sum of the weights shows a peak, the mean, the variance, and the deviation by performing statistical processing of the weights. A peak is a location with a local maximum value within a local range, and may be a point where a value actually exists, a point interpolated from discrete points where a value actually exists, or a peak on a curve approximating discrete points. Figure 6 shows a diagram representing the concept of a peak. In Figure 6, the horizontal axis represents the azimuth angle and elevation angle in three-dimensional space, and the vertical axis represents the sum of the weights. Although Figure 6 is two-dimensional, the horizontal axis conceptually represents the angle in three-dimensional space. The component with the largest peak value (maximum value) can be named the first peak, the component with the second largest peak value can be named the second peak, and so on, from largest to smallest peak value. The normal vector of the angle corresponding to the first peak can be called the first component vector, the normal vector of the angle corresponding to the second peak can be called the second component vector, and the normal vector of the angle corresponding to the third peak can be called the third component vector. Furthermore, if two peaks are close together and have a range, the mean or median of that range can be used as the peak position.

[0040] For example, if the first peak has the maximum weighted sum in the azimuth angle range of 12 to 13 degrees and the elevation angle range of 0 to 1 degree, then representative values ​​of the angle, such as the minimum, maximum, average, and median, are determined within that range. Then, the first component vector, which has the determined representative values ​​for the azimuth and elevation angles, is determined. If the angle is expressed as an integer rounded to the first decimal place, the integer angle represents the representative value.

[0041] The first peak represents the maximum total area of ​​triangles on the surface of the TIN model whose normal direction is oriented in the angular range corresponding to the first peak. Therefore, by extracting the first peak, it is possible to identify which direction of the faces constituting the 3D model has the largest total area. The first component vector is the normal vector corresponding to the face in the 3D model whose total area is largest when all faces are oriented in the same direction or within the same range.

[0042] Furthermore, the second peak represents the second largest local maximum, which is the sum of the areas of triangles whose normal direction is oriented in the angular range corresponding to the second peak, outside the region of the first peak. Therefore, by extracting the second peak, it is possible to identify which direction of the faces constituting the 3D model corresponds to the second largest local maximum. In this way, by performing statistical processing using the area of ​​each face of the 3D model as a weight, it is possible to analyze the direction and size of the faces of the 3D model.

[0043] The processing unit of the information processing device 1 determines the viewing direction and viewpoint of the 3D model based on the statistical values ​​obtained from statistical processing of the weights (S4). The viewing direction may be determined in such a way that attribute information can be correctly obtained when attribute information is obtained in S6. The viewing direction is set to the direction in which the 3D data is viewed from the outside, that is, the direction from the outside of the object toward the center. The viewing direction can be determined, for example, based on the direction of the first component vector. As an example, the viewing direction can be the direction in the opposite direction of the first component vector with the starting point being the center position of the 3D model. The center position of the 3D model can be determined as the center position of the rectangular prism (bounding box) surrounding the 3D model. In this way, the viewing direction can be determined to be the direction along the first component vector with the center of the 3D model as the starting point.

[0044] The viewpoint, which serves as the starting point of the line of sight, can be any position outside the bounding box surrounding the 3D model. However, it is preferable that the viewpoint is close to the 3D model, at a distance where the entire 3D model is included within the field of view when acquiring the image in S5.

[0045] Furthermore, when obtaining attribute information in S6, the viewing direction is not limited to the opposite direction of the first component vector, but may be a direction shifted from that direction, provided that the attribute information can be correctly obtained. The acceptable range of shift can be determined by the limit to which the surface corresponding to the first component vector is included in the image when obtaining a 2D image of the 3D model. This is because the surface corresponding to the first component vector occupies the largest surface area in the 3D model and is therefore good at representing the features and attributes of the 3D model.

[0046] The line of sight direction may also be determined based on the second or third component vector. For example, the line of sight direction can be the direction opposite to the second or third component vector with the starting point at the center position of the 3D model. Furthermore, the line of sight direction may be set to multiple directions.

[0047] The line of sight should be set so that the viewpoint is above the center of the 3D model (object). This is because 3D data is often calculated from images viewed from above, vertically above, and if the viewpoint is below the center, the image from that viewpoint will have less information, making it easy to produce an image with insufficient information.

[0048] Next, a 2D image of the 3D model as seen from the determined viewpoint and line of sight is obtained (S5). A plane perpendicular to the line of sight is placed between the viewpoint and the 3D model, and the image of the 3D model is projected onto that plane to obtain the 2D image.

[0049] Next, using a trained model that outputs attribute information from an input image, attribute information of a 3D model (object) is obtained from a 2D image viewed from the direction of the line of sight (S6). The trained model that outputs attribute information from an input image is a model that has been mechanically trained using a large number of 2D images and a large number of attribute information as training data, and is created by artificial intelligence (AI) such as a large-scale neural network. The trained model may be configured within the information processing device 1, or it may be provided by an external computer separate from the information processing device 1. External computers can take the form of servers or public clouds. In S6, when the 2D image acquired in S5 is input to the trained model that outputs attribute information from an input image, the trained model outputs attribute information such as the objects contained in the image. The output may be a description of the image or in the form of keywords.

[0050] If multiple viewing directions are set, multiple images are acquired from each viewing direction and these multiple images are input into the trained model. The trained model outputs attribute information corresponding to each image, but all of the output attribute information may be used as the object's attribute information, only the attribute information common to all images may be selected as the object's attribute information, or the attribute information may be determined by majority vote.

[0051] Next, attribute information output from the trained model is attached to the 3D model (object) (S7). Methods for attaching attribute information include, for example, tagging the 3D model or linking with a database. Attribute information can be tagged for each part of the 3D model corresponding to each attribute information and saved in a database or file format. For example, attribute data can be directly linked to each part of the 3D model. File formats include the IFC format for BIM data and Shapefile for GIS. With database linkage, attribute data related to 3D data can be saved in a separate database, and that attribute data can be retrieved when a specific part of the 3D data is selected.

[0052] (Example 1) As an example of a 3D model, Figure 7 shows a diagram of a 3D model of a building. In S3, when surface analysis of the 3D model is performed, it is calculated that the area of ​​the surface with a normal direction within the same angular range as the normal direction of the entrance surface shown in Figure 7 is the largest. Therefore, the first component vector with the same angular range as the normal direction of the entrance surface, starting from the center position of the 3D model, is determined, and the opposite direction of this first component vector is determined as the line of sight direction. In Figure 8, a bounding box BB that is tangent to and surrounds the 3D model is shown by a solid line. The line of sight direction VI is the opposite direction of the first component vector starting from the center position CP of the bounding box BB. The viewpoint VP can be determined so that the 3D model of the building is included within the field of view.

[0053] Figure 9 shows a 2D image of the 3D model as seen from the determined line of sight direction VI and viewpoint VP. The image in Figure 9 includes the front of the building, including the entrance. The image in Figure 9 is then input into the pre-trained model to obtain attribute information for the 3D model. The pre-trained model, with the image in Figure 9 as input, outputs attribute information such as a white entrance, windows, chimney, blue exterior walls, a brown roof, and stairs.

[0054] Figure 10 shows a 2D image of the 3D model viewed from the line of sight, where the line of sight is defined as the opposite direction of the second component vector starting from the center position CP of the rectangular prism BB. When the image in Figure 10 is input to the trained model, it outputs attribute information such as windows, chimney, blue exterior wall, and brown roof.

[0055] As shown in the example above, when comparing attribute information obtained from an image of a 3D model viewed from a viewing direction determined based on the first component vector with attribute information obtained from an image of a 3D model viewed from a viewing direction determined based on the second component vector, the first component vector yields more attribute information and more accurately represents the 3D model. Therefore, obtaining attribute information from only the image of the 3D model viewed from a viewing direction determined based on the first component vector requires less computation and less time to obtain accurate attribute information, compared to obtaining attribute information from 2D images of the 3D model viewed from each of multiple viewing directions.

[0056] As described above, according to the above embodiment, by using a trained model that outputs attribute information from input images, the effort and cost of inputting attribute information into a large number of 3D models through user operation in the software can be reduced, and input errors can be suppressed. Furthermore, in the above embodiment, by analyzing the surfaces of the 3D model, determining the viewing direction of the 3D model based on the analysis results, and acquiring a 2D image of the 3D model viewed from the determined viewing direction, an image can be obtained from which attribute information can be more accurately determined. In this way, even when using artificial intelligence and a trained model that outputs attribute information from input images for a wide variety of 3D models, the possibility of outputting incorrect attribute information is reduced, and attribute information can be determined more accurately. Therefore, correct attribute information can be attached to a wide variety of 3D data more easily, creating new methods of data utilization and improving the efficiency of data-driven work.

[0057] For example, by adding attributes such as windows and doors, as well as information about the location of windows and doors, to a 3D model of a building, in addition to information such as the building's height and roof, this information can be used for building maintenance, drone delivery services, autonomous driving services, and architectural planning.

[0058] (Comparative example) Next, we will explain a comparative example. In the comparative example, we will explain an example where the viewing direction of the 3D model of the building in Figure 7 is determined based on the normal direction of the faces of the rectangular prism BB that surrounds and touches the 3D model. The rectangular prism BB has six faces, and the normal directions of the six faces, i.e., the up and down, left and right, and front and back directions, can be set as the viewing direction. If the viewing direction is the opposite direction of the outward normal vector of the top surface of the rectangular prism BB, starting from the center position CP of the rectangular prism BB, the image of the 3D model viewed from a viewpoint at a certain distance will be as shown in Figure 11. When the image in Figure 11 is input into the trained model described above to obtain attribute information of the 3D model, the brown roof and chimney are output, but the windows, entrance, exterior walls and stairs are not output because they do not exist in the image.

[0059] If the line of sight is defined as the opposite direction of the outward normal vector of the bottom surface of the rectangular prism BB, starting from the center position CP of the rectangular prism BB, then the image of the 3D model viewed from a viewpoint at a certain distance will be as shown in Figure 12. In the image in Figure 12, the shape of the building's base can be recognized, but because actual measurement data is not available, the image is lacking in information. When the image in Figure 12 is input into the trained model described above, the information that it is a building is output, but detailed attribute information about the building's components is not output.

[0060] As another example, if a 3D object has an inclined surface such as a slope, setting a bounding box for the object will result in the object being viewed from the side of the slope when viewed from the side of the bounding box. In this way, 3D objects can be diverse, so if a bounding box is set for an object and the viewing direction is set based on the surface of the bounding box, the image viewed from that viewing direction may not have accurate attribute information. If the viewing direction is not set appropriately, even if an image of the 3D model is input into the trained model described above, it will not be possible to obtain appropriate attribute information.

[0061] Preferred embodiments of this disclosure have been described in detail above with reference to the attached drawings, but the technical scope of this disclosure is not limited to these examples. Furthermore, not all components shown in the embodiments are essential components of this disclosure. Also, features shown in each embodiment are applicable to other embodiments insofar as they do not contradict each other.

[0062] The information processing device 1 is not limited to a single device; it may consist of multiple distributed information processing devices, or it may be configured as a virtual server or container in the cloud.

[0063] When a 3D model is superimposed on an image representing 2D geographic information, the image viewed from the line of sight determined in S4 contains 2D geographic information. Therefore, when this image is input to the trained model described above, attribute information related to the 2D geographic information is output. For example, in the case of a 3D model of a building that is known to be close to the coastline from the geographic information, attribute information of "coastal building" is obtained. In this way, by superimposing a 3D model of an object onto geospatial information, attribute information can be obtained from an image of the 3D model containing geospatial information, viewed from the determined line of sight. [Industrial applicability]

[0064] This disclosure relates to an information processing method, an information processing device, and a program for assigning attribute information to a three-dimensional model, and has industrial applicability. [Explanation of Symbols]

[0065] 1. Information Processing Device

Claims

1. In an information processing method for assigning attribute information to a 3D model of an object, the computer, The surface area and normal direction of each surface of the three-dimensional model are calculated, and the line of sight direction is determined based on the normal direction weighted by the size of the surface area calculated for each surface. An image of the three-dimensional model viewed from the line of sight is obtained. The image viewed from the direction of the line of sight is input to a trained model that outputs attribute information from an input image to obtain attribute information for the 3D model. The attribute information is assigned to the aforementioned three-dimensional model. An information processing method characterized by the following:

2. The information processing method according to claim 1, characterized in that the line of sight direction is determined based on the total surface area of ​​the surfaces of the three-dimensional model that face the same range of directions.

3. The information processing method according to claim 2, characterized in that, for each surface of the three-dimensional model, the line of sight direction is determined based on the direction of the first component vector of the normal vector corresponding to the surface with the largest total surface area obtained by summing the surfaces facing the same direction.

4. The information processing method according to claim 3, characterized in that the direction along the first component vector, which starts from the center of the three-dimensional model, is determined as the line of sight direction.

5. The information processing method according to claim 1, characterized in that a surface area weight is added to the normal vector for each surface of the three-dimensional model, and the line of sight direction is determined based on the value of the weight of the normal vector.

6. The information processing method according to claim 5, characterized in that the line of sight direction is determined based on the sum of the weights of the normal vectors whose directions are within a predetermined range.

7. The information processing method according to claim 6, characterized in that the line of sight direction is determined based on the peak of the sum of the angles.

8. The information processing method according to claim 1, characterized in that when a specific part of the three-dimensional model to which attributes have been assigned is selected, attribute information of the specific part is output.

9. A program for causing a computer to execute the information processing method described in any one of claims 1 to 8.

10. In an information processing device that assigns attribute information to a three-dimensional model of an object, It has a processing unit, and the processing unit is The surface area and normal direction of each surface of the three-dimensional model are calculated, and the line of sight direction is determined based on the normal direction weighted by the size of the surface area calculated for each surface. An image of the three-dimensional model viewed from the line of sight is obtained. The image viewed from the direction of the line of sight is input to a trained model that outputs attribute information from an input image to obtain attribute information for the 3D model. An information processing device characterized by assigning the attribute information to the three-dimensional model.

Images