Information processing apparatus, information processing method, program, and information processing system

Abstract

To appropriately allow a user to recognize additional information to a subject part in a space image.SOLUTION: A server as an example of an information processing apparatus comprises a creation unit that displays a virtual space screen indicating a space, and creates a display screen displaying, superimposed on the virtual space screen, additional information to an article in the virtual space screen generated on the basis of the distance from the position of a viewpoint seeing the virtual space screen to the article.SELECTED DRAWING: Figure 15A

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, a program, and an information processing system. [Background technology]

[0002] Patent Document 1 discloses an information processing device 1 having an acquisition unit 121 that acquires an captured image generated by a terminal capturing an image of a target range and point cloud data indicating the three-dimensional positions of objects included in the target range measured by the terminal, an identification unit 124 that identifies an area corresponding to multiple reinforcing bars in the target range based on the point cloud data, a calculation unit 125 that calculates the spacing between two adjacent reinforcing bars among the multiple reinforcing bars based on the point cloud data, and a display control unit 122 that causes a display device to display the area corresponding to the multiple reinforcing bars by superimposing it on the captured image and also displays information indicating the spacing on the display device. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7150290 Summary of the Invention [Problem to be solved by the invention]

[0004] An object of the present invention is to allow a user to appropriately recognize additional information for a target portion in a spatial image. [Means for solving the problem]

[0005] The information processing device of the present invention displays a spatial image showing a space, and is equipped with a display screen generation means for generating a display screen that displays additional information for a target part, which is generated based on the distance from the viewpoint position for viewing the spatial image to the target part in the spatial image, superimposed on the spatial image. Effect of the Invention

[0006] According to the present invention, it is possible to allow a user to appropriately recognize additional information for a target portion in a spatial image. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram showing the overall configuration of a three-dimensional shape generation system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram showing the hardware configuration of the terminal device and the management server according to this embodiment. [Diagram 3] FIG. 3 is a functional block diagram of the three-dimensional shape generation system according to this embodiment. [Figure 4] FIG. 4 is a conceptual diagram showing an example of a setting information management table according to the present embodiment. [Diagram 5] FIG. 5 is a conceptual diagram showing an example of a virtual space management table according to the present embodiment. [Figure 6] FIG. 6 is a sequence diagram showing an example of the virtual space generation process according to the present embodiment. [Figure 7] FIG. 7 is a sequence diagram showing an example of the virtual space operation process according to this embodiment. [Figure 8] FIG. 8 is an explanatory diagram of the setting screen according to this embodiment. [Figure 9] FIG. 9 is a flowchart showing an example of a three-dimensional shape generation process according to this embodiment. [Figure 10] FIG. 10 is an explanatory diagram of the registration process according to this embodiment. [Figure 11] FIG. 11 is an explanatory diagram of the noise removal process according to this embodiment. [Figure 12] FIG. 12 is an explanatory diagram of the segmentation process according to this embodiment. [Figure 13] FIG. 13 is an explanatory diagram of the model matching and replacement process according to this embodiment. [Figure 14] FIG. 14 is an explanatory diagram of the operation screen according to this embodiment. [Figure 15A]FIG. 15A is a flowchart showing an example of the virtual space operation process according to this embodiment. [Figure 15B] FIG. 15B is a flowchart showing an example of a process for determining whether an article is protruding according to this embodiment. [Figure 16] FIG. 16 is an explanatory diagram of a three-dimensional image according to this embodiment. [Figure 17] FIG. 17 is an explanatory diagram of a virtual space screen according to this embodiment. [Figure 18] FIG. 18 is a second explanatory diagram of the virtual space screen according to this embodiment. [Figure 19] FIG. 19 is a third explanatory diagram of the virtual space screen according to this embodiment. [Figure 20] FIG. 20 is a fourth explanatory diagram of the virtual space screen according to this embodiment. [Figure 21] FIG. 21 is a fifth explanatory diagram of the virtual space screen according to this embodiment. [Figure 22] FIG. 22 is a fifth explanatory diagram of the virtual space screen according to this embodiment. [Diagram 23] FIG. 23 is a sixth explanatory diagram of the virtual space screen according to the present embodiment. [Figure 24] FIG. 24 is a seventh explanatory diagram of the virtual space screen according to this embodiment. [Diagram 25] FIG. 25 is an explanatory diagram of input information according to this embodiment. [Figure 26] FIG. 26 is an eighth explanatory diagram of the virtual space screen according to this embodiment. [Figure 27] FIG. 27 is a flowchart showing an example of the virtual space screen generating process when the designated additional information is included. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] In industries such as civil engineering and construction, the shift to BIM / CIM is being promoted in order to address issues such as the declining birthrate and aging population and to improve labor productivity.

[0009] BIM is an abbreviation for Building Information Modeling, and is a solution that uses a building database that adds attribute data such as cost, finish, and management information to a three-dimensional digital model of a building (hereafter referred to as a 3D model) created on a computer, allowing information to be utilized in every process from building design and construction to maintenance.

[0010] CIM is an abbreviation for Construction Information Modeling, and is a solution for the civil engineering field (general infrastructure such as roads, electricity, gas, and water) proposed following the example of BIM, which was being promoted in the construction field. Like BIM, it is being worked on as a way to improve the efficiency and sophistication of the entire construction production system by sharing information between related parties, mainly using 3D models.

[0011] A key aspect of promoting BIM / CIM is how to easily create 3D models of buildings and public facilities.

[0012] When building new buildings, it is relatively easy to convert to BIM / CIM because the finished product can be modeled from scratch using 3D CAD software. On the other hand, in the case of existing buildings, the hurdles to converting to BIM / CIM are higher because the original blueprints may no longer exist, or may differ from the blueprints due to renovations. Converting existing buildings to BIM in this way is called As-Build BIM, and is an important issue for promoting the future conversion to BIM / CIM.

[0013] One way to realize As-Build BIM is to use a laser scanner (hereafter referred to as LS) to measure the space and create a CAD model from the measured point cloud. Conventionally, the space was measured using photographs and tape measures, and sketched to restore the space, which resulted in a huge amount of work costs, but the introduction of LS has dramatically improved the efficiency of this work.

[0014] While As-Build using LS makes modeling easier, it also creates a task called point cloud processing, which did not exist in conventional work. In general point cloud processing, multi-point measurements are performed using LS, each point cloud is aligned, an integrated point cloud is constructed, unnecessary point clouds such as noise are removed, and finally, the point cloud is converted into a CAD model.

[0015] These processes are performed using the automatic model creation function of commercially available point cloud processing software. However, when different categories of objects, such as pipes, desks, and people, exist in the same space, i.e., in the same point cloud, attempting to generate the three-dimensional shapes of these different categories of objects all at once may result in insufficient accuracy or long processing times.

[0016] On the other hand, generating a three-dimensional shape of an object in a single category requires repeated generation for each required category, which is time-consuming.

[0017] In view of the above problems, the present embodiment aims to appropriately generate a three-dimensional shape with the accuracy required by the user without requiring more processing time and effort than necessary.

[0018] Another issue is how to utilize the constructed BIM / CIM. Specifically, the three-dimensional virtual space restored by BIM / CIM can be used for design and construction purposes, as well as other work such as maintenance and site inspections. In other words, in addition to using it as a blueprint, other uses can be considered, such as keeping records in the three-dimensional virtual space and sharing it with others. However, compared to two-dimensional space, three-dimensional virtual space contains a lot more information, making it difficult for first-time viewers to handle.

[0019] Another object of the present embodiment is to make the three-dimensional virtual space screen easier to use by appropriately displaying additional information for objects in the three-dimensional virtual space screen.

[0020] 1 is a diagram showing the overall configuration of a three-dimensional shape generating system according to an embodiment of the present invention. The three-dimensional shape processing system 1 of this embodiment is constructed by a terminal device 3, which is an example of an input / output device, and a server 5.

[0021] The server 5 is an example of an information processing device connected to an input / output device via a network. The server 5 is also an example of a three-dimensional shape generating device that generates three-dimensional shape information indicating a three-dimensional shape corresponding to a three-dimensional point group by using model shape information indicating a three-dimensional model shape.

[0022] Here, a three-dimensional point cloud is a collection of coordinate points in a virtual three-dimensional space that can be handled by a computer or the like. A three-dimensional point cloud is also called a point cloud. A three-dimensional point cloud is a collection of coordinate points that correspond to measurement points on the surface of an object when a space containing the object is measured using a laser scanner LS or the like. In addition, color information may be added to each coordinate point, and the RGB value of each coordinate point may be added as color information.

[0023] Although the example shows that the 3D point cloud is measured using a laser scanner LS, other optical measurement means or mechanical measurement means may be used. Optical measurement means include a method using a stereo camera or a method using Visual SLAM.

[0024] Furthermore, three-dimensional shape information is information that indicates the three-dimensional shape of an object that can be handled by a computer or the like. Information that indicates a three-dimensional shape is information that can specify the three-dimensional shape geometrically; for example, in the case of a sphere, the coordinates of the center and the radius correspond to information that indicates the three-dimensional shape. When the three-dimensional shape of an object is expressed as a polyhedron (polygon), the coordinate points of each vertex of the polyhedron are an example of information that indicates the three-dimensional shape. In addition, any information that can uniquely define the shape of the object can be used as information that represents the three-dimensional shape.

[0025] The three-dimensional shape information may include information related to the color and material of the object in addition to information indicating the three-dimensional shape of the object.

[0026] The three-dimensional model shape is a model such as a template used to generate three-dimensional shape information from a three-dimensional point group. The model shape information is information indicating the three-dimensional model shape, and one piece of model shape information corresponds to one three-dimensional model shape.

[0027] The model shape information may include information related to the color and material of the three-dimensional model in addition to the three-dimensional model shape. Specifically, the three-dimensional model shape showing a plane may have information called texture, such as the color and pattern of the wall, and information on the material.

[0028] Texture information makes it easy to create CAD models with colors and patterns. Material information can also be passed directly to CAD models that can have material information, eliminating the need for users to make settings.

[0029] Here, "texture" originally refers to the feel and texture of an object's surface, and in three-dimensional shape information, it refers to a pattern or image that is affixed to the surface of a three-dimensional shape in order to express the surface feel of an object.

[0030] In addition, CAD models are generally created using "surfaces," "solids," and "polygons." However, "solids" contain not only the appearance but also information about the contents such as material, making it possible to calculate mass and volume and to represent cross-sectional shapes, making them the closest representation of an actual object.

[0031] Furthermore, setting color, material, brightness, background, etc. on a CAD model is called "rendering," and by rendering, you can create beautiful images like photographs.

[0032] As described above, the three-dimensional shape information includes three-dimensional geometric shapes handled by 3DCAD, BIM, and CIM, or geometric information, mesh objects, polygon information with texture, and the like.

[0033] The terminal device 3 and the server 5 can communicate with each other via a communication network 100. The communication network 100 is constructed by the Internet, a mobile communication network, a LAN (Local Area Network), and the like. The communication network 100 may include not only wired communication but also wireless communication networks such as 3G (3rd Generation), WiMAX (Worldwide Interoperability for Microwave Access), and LTE (Long Term Evolution). The terminal device 3 can communicate with each other using a short-range communication technology such as NFC (Near Field Communication) (registered trademark).

[0034] <Hardware configuration> 2 is a diagram showing the hardware configuration of the terminal device and the management server according to this embodiment. Each piece of hardware configuration of the terminal device 3 is indicated by a reference number in the 300s. Each piece of hardware configuration of the server 5 is indicated by a hardware number in the 500s in parentheses.

[0035] The terminal device 3 includes a CPU (Central Processing Unit) 301, a ROM (Read Only Memory) 302, a RAM (Random Access Memory) 303, a HD (Hard Disk) 304, a HDD (Hard Disk Drive) 305, a recording medium 306, a media I / F 307, a display 308, a network I / F 309, a keyboard 311, a mouse 312, a CD-RW (Compact Disc-ReWritable) drive 314, and a bus line 310.

[0036] Among these, the CPU 301 controls the operation of the entire terminal device 3. The ROM 302 stores a program used to drive the CPU 301. The RAM 303 is used as a work area for the CPU 301. The HD 304 stores various data such as programs. The HDD 305 controls reading or writing of various data from the HD 304 under the control of the CPU 301. The media I / F 307 controls reading or writing (storing) of data from a recording medium 306 such as a flash memory. The display 308 displays various information such as a cursor, a menu, a window, characters, or an image. The network I / F 309 is an interface for data communication using the communication network 100. The keyboard 311 is a type of input means having a plurality of keys for inputting characters, numbers, various instructions, and the like. The mouse 312 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, and the like. The CD-RW drive 314 controls reading or writing of various data from a CD-RW 313 as an example of a removable recording medium.

[0037] The server 5 also includes a CPU 501, a ROM 502, a RAM 503, a HD 504, a HDD 505, a recording medium 506, a media I / F 507, a display 508, a network I / F 509, a keyboard 511, a mouse 512, a CD-RW drive 514, and a bus line 510. These have the same configuration as the above-mentioned configurations (CPU 301, ROM 302, RAM 303, HD 304, HDD 305, recording medium 306, media I / F 307, display 308, network I / F 309, keyboard 311, mouse 312, CD-RW drive 314, and bus line 310), and therefore description thereof will be omitted.

[0038] Note that a CD-R drive or the like may be used instead of the CD-RW drive 314 (514). The terminal device 3 and the server 5 may each be constructed by a single computer, or may be constructed by multiple computers in which each unit (function, means, or storage unit) is divided and assigned arbitrarily.

[0039] FIG. 3 is a functional block diagram of the three-dimensional shape generation system according to this embodiment.

[0040] As shown in Fig. 3, the terminal device 3 has a transmission / reception unit 31, a reception unit 32, a display control unit 34, a determination unit 36, and a storage / readout unit 39. Each of these units is a function or a means for performing a function that is realized when any of the components shown in Fig. 2 operates in response to an instruction from the CPU 301 in accordance with a program loaded from the HD 304 onto the RAM 303. The terminal device 3 also has a storage unit 3000 constructed by the RAM 303 and HD 304 shown in Fig. 2.

[0041] (Functional configuration of terminal device) Next, each component of the terminal device 3 will be described.

[0042] The transmission / reception unit 31 is an example of a transmission means, and is realized by instructions from the CPU 301 shown in FIG. 2 and the network I / F 309, and transmits and receives various data (or information) with other terminals, devices or systems via the communication network 100.

[0043] The reception unit 32 is an example of a reception means, and is realized mainly by commands from the CPU 301 shown in FIG. 2, as well as the keyboard 311 and the mouse 312, and receives various inputs from the user.

[0044] The display control unit 34 is an example of a display control means, and is realized by instructions from the CPU 301 shown in FIG. 2, and causes the display 308, which is an example of a display unit, to display various images and screens.

[0045] The determination unit 36 ​​is an example of a determination means, and is realized by instructions from the CPU 301 shown in FIG. 2, and performs various determinations.

[0046] The memory / read unit 39 is an example of a memory control means, and is executed by instructions from the CPU 301 shown in FIG. 2, as well as the HDD 305, the media I / F 307, and the CD-RW drive 314, to store various types of data in the memory unit 3000, the recording medium 306, and the CD-RW 313, and to read various types of data from the memory unit 3000, the recording medium 306, and the CD-RW 313.

[0047] <Management Server Functionality Configuration> The server 5 has a transmission / reception unit 51, a processing unit 53, a determination unit 55, a generation unit 56, a setting unit 57, and a storage / reading unit 59. Each of these units is a function or a means for performing a function that is realized when any of the components shown in Fig. 2 operates in response to an instruction from the CPU 501 in accordance with a program loaded from the HD 504 onto the RAM 503. The server 5 also has a memory unit 5000 constructed by the HD 504 shown in Fig. 2. The memory unit 5000 is an example of a memory means.

[0048] (Management server functional configuration) Next, each component of the server 5 will be described. The server 5 may be configured to distribute each function among multiple computers. Furthermore, the server 5 will be described as a server computer that exists in a cloud environment, but may be a server that exists in an on-premise environment.

[0049] The transmission / reception unit 51 is an example of a transmission means, and is realized by instructions from the CPU 501 shown in FIG. 2 and the network I / F 509, and transmits and receives various data (or information) with other terminals, devices or systems via the communication network 100.

[0050] The processing unit 53 is an example of a processing means, and is realized by instructions from the CPU 501 shown in Fig. 2, and performs various processes described below. The processing unit 53 is an example of a three-dimensional information generating means that generates three-dimensional shape information.

[0051] The determination unit 55 is an example of a determination means, and is realized by instructions from the CPU 501 shown in FIG. 2, and performs various determinations described later.

[0052] The generating unit 56 is an example of a generating means, and is realized by instructions from the CPU 501 shown in FIG. 2, and generates various screens.

[0053] The setting unit 57 is an example of a setting means, and is realized by commands from the CPU 501 shown in FIG. 2, and performs various settings and determinations, which will be described later.

[0054] 2, and the HDD 505, the media I / F 507, and the CD-RW drive 514, and performs processes such as storing various data in the storage unit 5000, the recording medium 506, and the CD-RW 513, and reading various data from the storage unit 5000, the recording medium 506, and the CD-RW 513. The storage unit 5000, the recording medium 506, and the CD-RW 513 are examples of storage means.

[0055] The memory unit 5000 contains a setting information management DB 5001 composed of a setting information management table, a memory processing management DB 5002, a point cloud management DB 5003, a three-dimensional shape management DB 5004, a model shape management DB 5005, and a virtual space management DB 5006 composed of a virtual space management table.

[0056] The setting information management DB 5001 stores and manages various information, the storage processing management DB 5002 stores and manages various processing programs for generating three-dimensional shapes, the point cloud management DB 5003 stores and manages three-dimensional point cloud information for generating three-dimensional shapes, and the three-dimensional shape management DB 5004 stores and manages three-dimensional shape information generated using model shape information. The model shape management DB 5005 stores and manages model shape information indicating three-dimensional model shapes for each model identification information, and the virtual space management DB 5006 stores and manages information on virtual spaces including three-dimensional shapes.

[0057] FIG. 4 is a conceptual diagram showing an example of a setting information management table according to the present embodiment.

[0058] The setting information management table is a table for managing three-dimensional point cloud data for generating a three-dimensional shape and a history of a process for generating the three-dimensional shape. A setting information management DB 5001 configured by a setting information management table as shown in Fig. 4 is constructed in the storage unit 5000. In this setting information management table, the file name of the three-dimensional point cloud data, the history of a process for generating the three-dimensional shape, and attribute information indicating the attributes of the user are associated and managed for each user ID, which is an example of user information.

[0059] FIG. 5 is a conceptual diagram showing an example of a virtual space management table according to the present embodiment.

[0060] The virtual space management table is a table for managing information about a virtual space including three-dimensional shape information. A virtual space management DB 5006 is constructed in the storage unit 5000, which is configured by a virtual space management table as shown in FIG.

[0061] In this virtual space management table, model identification information, position information, item information, attribute information, dimension line position information, and dimension information are associated with each other and managed for each virtual space identification information that identifies a virtual space.

[0062] The model identification information is information for identifying model shape information that indicates the three-dimensional model shape, and the position information is information that indicates the position of the three-dimensional model shape in the virtual space using three-dimensional coordinates of XYZ.

[0063] The article information is information about an article represented by a three-dimensional shape composed of a single model shape or a plurality of model shapes. The article represented by the three-dimensional shape is an example of a target part in a space image representing a space.

[0064] The attribute information is information for identifying the attributes of the user who operates the three-dimensional shape displayed on the display 308 .

[0065] Dimension line position information is information that specifies the position information of dimension lines for a specific three-dimensional shape in three-dimensional coordinates of XYZ. Dimension lines include unidirectional lines that indicate one dimension such as (Xa, Ya, Za) to (Xb, Yb, Zb), bidirectional lines that indicate two dimensions such as (X1, Y1, Z1) to (X2, Y2, Z2) and (X3, Y3, Z3) to (X4, Y4, Z4), and three-directional lines that indicate three dimensions.

[0066] The dimension information is information that indicates, as a numerical value, the dimension corresponding to the dimension line. If the dimension line is a line in one direction, the length is indicated as a numerical value, if the dimension line is a line in two directions, at least one of the length and the area is indicated as a numerical value, and if the dimension line is a line in three directions, at least one of the volume or capacity, the length and the area is indicated as a numerical value. If the dimension information is area, the target surface may be solid-colored instead of the dimension line. The dimension line position information and the dimension line are examples of additional information for the target part in the spatial image.

[0067] FIG. 6 is a sequence diagram showing an example of a three-dimensional shape generation process according to this embodiment.

[0068] The reception unit 32 of the terminal device 3 receives an input operation related to the user ID of the user on the input / output screen displayed on the display 308 (step S1). The transmission / reception unit 31 transmits a request for a setting screen including the user ID received in step S1 to the server 5, and the transmission / reception unit 51 of the server 5 receives the request transmitted from the terminal device 3 (step S2).

[0069] Next, the memory / read unit 59 of the server 5 searches the setting information management DB 5001 using the user ID included in the request received in step S2 as a search key, to read out the file name of the three-dimensional point cloud data associated with the user ID included in the request and the history of the process that generated the three-dimensional shape, and the generation unit 56 of the server 5 generates a setting screen based on the file name and history read by the memory / read unit 59 (step S3).

[0070] This setting screen includes a model information setting screen that accepts model setting operations for setting multiple pieces of model shape information used to generate three-dimensional shape information from multiple pieces of model shape information, a point cloud setting screen that accepts point cloud setting operations for setting a three-dimensional point cloud used to generate three-dimensional shape information, and a processing setting screen that accepts processing setting operations for setting a processing program for generating three-dimensional shape information.

[0071] The processing setting screen includes a display that is initially set based on the history read by the memory / read unit 59 .

[0072] The transmitting / receiving unit 51 transmits setting screen information relating to the setting screen generated in step S3 to the terminal device 3, and the transmitting / receiving unit 31 of the terminal device 3 receives the setting screen information transmitted from the server 5 (step S4). Step S4 is an example of a transmitting step of transmitting model information setting screen information relating to the model information setting screen, and the transmitting / receiving unit 31 is an example of a transmitting means.

[0073] Next, the display control unit 34 of the terminal device 3 causes the display 308 to display the setting screen received in step S4 (step S5). The reception unit 32 of the terminal device 3 receives a predetermined input operation by the user on the displayed setting screen. This input operation includes a model setting operation for setting multiple pieces of model shape information used to generate three-dimensional shape information from multiple pieces of model shape information, a point cloud setting operation, and a process setting operation. Step S5 is an example of a reception step for receiving a model setting operation for setting multiple pieces of model shape information used to generate three-dimensional shape information from multiple pieces of model shape information.

[0074] The transmitting / receiving unit 31 transmits input information related to the input operation received by the receiving unit 32 to the server 5, and the transmitting / receiving unit 51 of the server 5 receives the input information transmitted from the terminal device 3 (step S6). This input information includes model setting information for setting a plurality of pieces of model shape information used to generate three-dimensional shape information from a plurality of pieces of model shape information, point cloud setting information, and processing setting information.

[0075] The memory / read unit 59 of the server 5 updates the history of the process that generated the three-dimensional shape, which is associated with the user ID and stored in the setting information management DB 5001, based on the processing setting information and model setting information contained in the input information received in step S6 (step S7).

[0076] Next, the memory / read unit 59 of the server 5 searches the point cloud management DB 5003 using the point cloud setting information included in the input information received in step S6 as a search key, and reads out the three-dimensional point cloud data associated with the point cloud setting information.

[0077] Furthermore, the storage / reading unit 59 searches the stored processing management DB 5002 using the processing setting information included in the input information received in step S6 as a search key, thereby reading out the processing program associated with the processing setting information.

[0078] Then, the storage / reading unit 59 searches the model shape management DB 5005 using the model identification information corresponding to the model setting information included in the input information received in step S6 as a search key, thereby reading out the three-dimensional model shape associated with the model identification information.

[0079] The processing unit 53 of the server 5 generates three-dimensional shape information based on the three-dimensional point cloud data, the processing program, and the three-dimensional model shape read from the storage / read unit 59 (step S8).

[0080] Step S8 is an example of a three-dimensional information generation step in which three-dimensional shape information is generated using a plurality of model shape information set by the model setting operation based on a model setting operation on the model information setting screen 1230 and point cloud information indicating a three-dimensional point cloud.

[0081] The generating unit 56 of the server 5 generates an operation screen for receiving an operation input for generating three-dimensional shape information, and the transmitting / receiving unit 51 transmits operation screen information relating to the operation screen to the terminal device 3 (step S9).

[0082] The transmitter / receiver 31 of the terminal device 3 receives the operation screen information transmitted from the server 5, the display control unit 34 of the terminal device 3 displays the received operation screen on the display 308, and the reception unit 32 of the terminal device 3 receives a predetermined input operation by the user on the displayed operation screen (step S10).

[0083] This input operation includes a shape setting operation for setting one piece of three-dimensional shape information from a plurality of pieces of three-dimensional shape information.

[0084] The transmitting / receiving unit 31 transmits input information relating to the input operation accepted by the accepting unit 32 to the server 5, and the transmitting / receiving unit 51 of the server 5 receives the input information transmitted from the terminal device 3 (step S11).

[0085] This input information includes shape setting information obtained by a shape setting operation, and the processing unit 53 of the server 5 determines three-dimensional shape information based on the shape setting information included in the input information received in step S11.

[0086] The terminal device 3 and the server 5 repeatedly execute steps S8 to S11 as necessary.

[0087] The processing unit 53 converts the generated three-dimensional shape information into a CAD format or the like, and the storage / reading unit 59 stores the converted three-dimensional shape information in the three-dimensional shape management DB 5004, the recording medium 506, or the CD-RW 513 in association with the model identification information of each of the single or multiple model shapes used to generate the three-dimensional shape and the position information in the three-dimensional point cloud data read in step S8 (step S12). The CAD format is three-dimensional shape information that can be handled by 3D CAD. When the three-dimensional shape information is to be handled by a commercially available 3D CAD, the three-dimensional shape information is converted into the 3D CAD format.

[0088] The memory / read unit 59 updates the history of the process that generated the three-dimensional shape, which is associated with the user ID and stored in the setting information management DB 5001, based on the model shape information corresponding to the determined three-dimensional shape information (step S13).

[0089] Next, the storage / readout unit 59 of the server 5 reads out all the three-dimensional shape information generated based on the three-dimensional point cloud data read out in step S8 from the three-dimensional shape management DB 5004 together with the model identification information and position information, and the generation unit 56 of the server 5 generates a virtual space screen including the three-dimensional shapes read out by the storage / readout unit 59 (step S14). This virtual space screen is a screen that shows the space related to the three-dimensional point cloud data read out in step S8 as a virtual space including one or more three-dimensional shapes generated based on the three-dimensional point cloud data.

[0090] The transmitting / receiving unit 51 transmits the virtual space screen information indicating the virtual space screen generated in step S14 to the terminal device 3 (step S15).

[0091] The transmitter / receiver 31 of the terminal device 3 receives the virtual space screen information transmitted from the server 5, and the display control unit 34 of the terminal device 3 causes the display 308 to display the virtual space screen indicated by the received virtual space screen information (step S16).

[0092] The accepting unit 32 of the terminal device 3 accepts a predetermined input operation by the user regarding the displayed virtual space screen. The input operation includes an operation for setting the virtual space identification information, the item information, the usage information, the attribute information, the operation type, the operation information, and the recommendation information shown in FIG.

[0093] The transmitting / receiving unit 31 transmits input information related to the input operation accepted by the accepting unit 32 to the server 5, and the transmitting / receiving unit 51 of the server 5 receives the input information transmitted from the terminal device 3 (step S17). This input information includes virtual space identification information, item information, usage information, attribute information, operation type, operation information, and recommendation information.

[0094] The storage / reading unit 59 of the server 5 associates the virtual space identification information, item information, use information, attribute information, operation type, operation information, and recommendation information contained in the input information received in step S17 with the model identification information and position information read in step S14, and stores them in the virtual space management DB (step S18). As a result, the various information shown in FIG. 5 is stored in the virtual space management DB 5006.

[0095] In the above, the functions of the server 5 in FIG. 3 may be integrated into the terminal device 3, and the processes of the server 5 in FIG.

[0096] FIG. 7 is a sequence diagram showing an example of the virtual space operation process according to this embodiment.

[0097] The accepting unit 32 of the terminal device 3 accepts an input operation related to virtual space identification information on the input / output screen displayed on the display 308 (step S101).

[0098] The transmission / reception unit 31 transmits to the server 5 virtual space identification information related to the input operation accepted by the acceptance unit 32 in step S101, and the transmission / reception unit 51 of the server 5 receives the virtual space identification information transmitted from the terminal device 3 (step S102).

[0099] The memory / read unit 59 of the server 5 searches the virtual space management DB 5006 using the virtual space identification information received in step S102 as a search key, thereby reading out the model identification information, position information and item information associated with the virtual space identification information, and also searches the model shape management DB 5005 using the model identification information read from the virtual space management DB 5006 as a search key, thereby reading out the three-dimensional model shape associated with the model identification information.

[0100] The generation unit 56 of the server 5 generates a virtual space screen based on the model identification information, the three-dimensional model shape, the position information, and the item information read by the storage / reading unit 59 (step S103).

[0101] The transmitting / receiving unit 51 transmits the virtual space screen information indicating the virtual space screen generated in step S103 to the terminal device 3 (step S104).

[0102] The transmission / reception unit 31 of the terminal device 3 receives the virtual space screen information transmitted from the server 5, and the display control unit 34 of the terminal device 3 causes the display 308 to display the virtual space screen indicated by the received virtual space screen information (step S105).

[0103] The accepting unit 32 of the terminal device 3 accepts a predetermined input operation by the user regarding the displayed virtual space screen and an operation regarding a three-dimensional shape included in the virtual space screen. The input operation includes an operation of inputting a user ID and usage information.

[0104] The transmitting / receiving unit 31 transmits input information related to the input operation received by the receiving unit 32 to the server 5, and the transmitting / receiving unit 51 of the server 5 receives the input information transmitted from the terminal device 3 (step S106). The input information includes a user ID, designation information for designating the type of dimension information, operation information for changing the position and orientation of a virtual camera located at the viewpoint position for viewing the virtual space, and designation information for designating an item.

[0105] The storage / reading unit 59 of the server 5 searches the setting information management DB 5001 using the user ID received in step S106 as a search key to read out attribute information associated with the user ID, and searches the model shape management DB 5005 using the item information, designation information, and specific information received in step S106 and the attribute information read from the setting information management DB 5001 as search keys to read out dimension line position information and dimension information associated with the item information, designation information, specific information, and attribute information.

[0106] The generation unit 56 of the server 5 generates a virtual space screen in which the dimension line position information and the dimension information read by the storage / read unit 59 are superimposed on the virtual space screen generated in step S103 and with the viewpoint changed based on the operation information received in step S106 (step S107).

[0107] The additional information such as dimension line position information and dimension information is not limited to information stored in association with attribute information, but may be information calculated based on three-dimensional shape information. For example, the processing unit 53 can calculate the dimension line position information based on the three-dimensional shape information. The processing unit 53 can also calculate the dimension information based on the read dimension line position information or the calculated dimension line position information.

[0108] The transmitting / receiving unit 51 transmits the virtual space screen information indicating the virtual space screen generated in step S107 to the terminal device 3 (step S108).

[0109] The transmitting / receiving unit 31 of the terminal device 3 receives the virtual space screen information transmitted from the server 5, and the display control unit 34 of the terminal device 3 causes the display 308 to display the virtual space screen indicated by the received virtual space screen information (step S109). The above steps S106 to S109 are repeatedly executed.

[0110] In the above, the functions of the server 5 in FIG. 3 may be integrated into the terminal device 3, and the processes of the server 5 in FIG.

[0111] FIG. 8 is an explanatory diagram of a display screen according to this embodiment.

[0112] FIG. 8 shows a display screen 1000 displayed on the display 308 of the terminal device 3 in step S5 of the sequence diagram shown in FIG.

[0113] The display control unit 34 of the terminal device 3 causes the display screen 1000 to display a user information display screen 1100, a setting screen 1200, and a confirmation button 1300, which is an example of an instruction acceptance screen.

[0114] The setting screen 1200 includes a point cloud setting screen 1210, a processing setting screen 1220, and a model information setting screen 1230 (an example of a reception screen).

[0115] The point cloud setting screen 1210 is a screen for receiving a point cloud setting operation for setting point cloud information indicating a three-dimensional point cloud used for generating three-dimensional shape information, and the display control unit 34 displays point cloud setting boxes 1212, 1214 in association with the respective file names of the multiple point cloud data read by the storage / reading unit 59. A plurality of point cloud setting boxes 1212, 1214 can be set.

[0116] The processing setting screen 1220 is a screen that accepts processing setting operations for setting a processing program for generating three-dimensional shape information, and the display control unit 34 displays processing setting boxes 1222, 1224, and 1226 in association with the names of a plurality of processes, respectively.

[0117] 8, a process setting box 1222 is for setting a process program related to registration, a process setting box 1224 is for setting a process program related to noise removal, and a process setting box 1226 is for setting a process program related to segmentation. Details of these process programs will be described later. Here, the process setting box 1222 related to registration may be set automatically when multiple point clouds are set on the point cloud setting screen 1210. Also, the process related to segmentation may be always executed, and the process setting box 1226 related to segmentation may be omitted.

[0118] The display control unit 34 initially sets and displays the settings in the process setting boxes 1222, 1224, and 1226 based on the history read by the storage / readout unit 59. If there is no history, a predetermined setting is displayed as the initial setting.

[0119] The model information setting screen 1230 is an example of a reception screen that receives a model setting operation for setting a plurality of pieces of model shape information used to generate three-dimensional shape information from a plurality of pieces of model shape information.

[0120] The model information setting screen 1230 includes model information setting boxes 1231 , 1234 , 1235 , and 1236 , model candidate setting boxes 1232 and 1233 , and a model display screen 1240 .

[0121] The display control unit 34 displays model information setting boxes 1231, 1234, and 1235 in association with the names of the multiple pieces of model information, and displays model candidate setting boxes 1232 and 1233 in association with the names of the multiple model shapes. Here, model information indicates a group of model shape information including one or more pieces of model shape information.

[0122] In FIG. 8, model information setting box 1231 is for setting model information indicating the three-dimensional model shape of piping, model information setting box 1234 is for setting model information indicating the three-dimensional model shape of a desk, and model information setting box 1235 is for setting model information indicating the three-dimensional model shape of a person.

[0123] In addition, the model candidate setting box 1232 is for setting model shape information indicating the three-dimensional model shape of the piping model A, and the model candidate setting box 1233 is for setting model shape information indicating the three-dimensional model shape of the piping model B.

[0124] The display control unit 34 displays the model candidates 1242, 1243, or the model shape 1244 on the model display screen 1240 in association with the respective names of the set model information or model candidates. If the model shape information includes color, it may be displayed in color. Here, each of the multiple model candidates includes a three-dimensional model shape having a different shape from each other, and is used to generate multiple pieces of three-dimensional shape information from the same region of the three-dimensional point cloud, and the generated multiple pieces of three-dimensional shape information are determined as one piece of three-dimensional shape information in a subsequent process.

[0125] A model candidate 1242 indicates a plurality of three-dimensional model shapes of a piping model A, a model candidate 1243 indicates a plurality of three-dimensional model shapes of a piping model B, and a model shape 1244 indicates a three-dimensional model shape of a desk.

[0126] When a setting box is pointed to with a pointing device such as the mouse 312, the display control unit 34 of the reception unit 32 of the terminal device 3 displays a check mark in the setting box, and the reception unit 32 accepts the various setting operations. When the confirm button 1300 is operated, the reception unit 32 confirms the various setting operations.

[0127] Then, as described in step S6 of FIG. 6, the transmitting / receiving unit 31 transmits to the server 5 input information including various setting information based on the various setting operations received by the receiving unit 32.

[0128] FIG. 9 is a flowchart showing an example of a three-dimensional shape generation process according to this embodiment, and shows a process corresponding to step S8 in FIG.

[0129] The processing unit 53 of the server 5 acquires the three-dimensional point cloud data read from the storage / reading unit 59 (step S21), and if multiple three-dimensional point cloud data are acquired and the processing program read from the storage / reading unit 59 includes a processing program related to registration, executes registration processing (step S22). The registration processing is processing for converting multiple three-dimensional point clouds into one integrated three-dimensional point cloud.

[0130] When the processing programs read from the storage / readout unit 59 include a processing program related to noise removal, the processing unit 53 executes noise removal processing (step S23). The noise removal processing is processing for removing unnecessary points from a three-dimensional point cloud.

[0131] The setting unit 57 of the server 5 sets the model information based on the model setting information included in the input information received in step S6 of Fig. 6 (step S24). For example, as shown in Fig. 8, when the model setting information includes model information indicating a three-dimensional model shape of a pipe and model information indicating a three-dimensional model shape of a desk, the setting unit 57 first sets the model information indicating the three-dimensional model shape of the pipe.

[0132] The setting unit 57 sets one or more pieces of model shape information based on the model setting information included in the input information received in step S6 in Fig. 6 (step S25). For example, as shown in Fig. 8, when the model setting information includes model shape information indicating the three-dimensional model shape of the piping model A and model shape information indicating the three-dimensional model shape of the piping model B as multiple model candidates, the setting unit 57 first sets the model shape information indicating the three-dimensional model shape of the piping model A.

[0133] If the processing programs read from the storage / read unit 59 include a processing program related to segmentation, the processing unit 53 executes a segmentation process (step S26).

[0134] Segmentation processing is also called point cloud classification processing, and techniques such as machine learning PointTransformer are known. (Note) X.-F. Han, Y.-J. Kuang, and G.-Q. Xiao, ''Point cloud learning with transformer,'' 2021, arXiv:2104.13636.

[0135] The segmentation process is a process of labeling specific points in a three-dimensional point cloud so that they can be distinguished from other point clouds, and may be performed in conjunction with a clustering process that groups close points among the labeled points.

[0136] The processing unit 53 compares the point group labeled in step S26 with each of the multiple three-dimensional model shapes included in the model shape information set in step S25, and replaces the specific area with the three-dimensional model shape that is the closest in shape (step S27). If there is no optimal model shape, the processing unit 53 may adjust the dimensions and shape of the model shape. If there is a possibility that the texture information is different for the same shape, the processing unit 53 may also compare models that are the same shape but have different textures. The processing unit 53 may also omit the segmentation process in step S26 and execute step S27 after step S25.

[0137] The determination unit 55 of the server 5 determines whether there is a point cloud for which replacement has not been processed (step S28), and if there is an unprocessed point cloud, the process returns to step 27, and the processing unit 53 executes a process of replacing a specific area with respect to the unprocessed point cloud by using the three-dimensional model shape for the unprocessed point cloud. If the ratio of the unprocessed point cloud to all point clouds becomes equal to or less than a predetermined value, the determination unit 55 may determine that there is no unprocessed point cloud, that is, that the processing of the point clouds is completed.

[0138] On the other hand, if there is no unprocessed point cloud, the determination unit 55 determines whether there is an unprocessed model candidate (step S29). If there is an unprocessed model candidate, the process returns to step 25, and the setting unit 57 sets model shape information included in the unprocessed model candidate. For example, the setting unit 57 sets model shape information indicating the three-dimensional model shape of the piping model B following model shape information indicating the three-dimensional model shape of the piping model A as multiple model candidates.

[0139] When there are no unprocessed model candidates, the processing unit 53 determines three-dimensional shape information based on the shape setting information included in the input information received in step S11 of Fig. 6 (step S30). For example, the setting unit 57 determines one of the three-dimensional shape information generated based on the model shape information of the piping model A and the three-dimensional shape information generated based on the model shape information of the piping model B as the three-dimensional shape information as the multiple model candidates.

[0140] The determination unit 55 determines whether there is unprocessed model information (step S31), and if there is no unprocessed model information, ends the process.

[0141] If there is unprocessed model information, the process returns to step 24, and the setting unit 57 sets the unprocessed model information. For example, the setting unit 57 sets the model information of the desk following the model information of the piping.

[0142] FIG. 10 is an explanatory diagram of the registration process according to this embodiment.

[0143] FIG. 10( a ) shows a first three-dimensional point cloud 1410 , and FIG. 10( b ) shows a second three-dimensional point cloud 1420 .

[0144] As an example, the processing unit 53 converts the first three-dimensional point cloud 1410 and the second three-dimensional point cloud 1420 into a single integrated three-dimensional point cloud by aligning feature points 1410a contained in the first three-dimensional point cloud 1410 with feature points 1420a contained in the second three-dimensional point cloud 1420.

[0145] 11 is an explanatory diagram of the noise removal process according to this embodiment. The processing unit 53 removes unnecessary points 1440 from a three-dimensional point group 1430.

[0146] FIG. 12 is an explanatory diagram of the segmentation process according to this embodiment.

[0147] The processing unit 53 labels certain points in the three-dimensional point cloud 1430 to form a labeled point cloud 1450 (black circles) so that they can be distinguished from other point clouds.

[0148] FIG. 13 is an explanatory diagram of the model matching and replacement process according to this embodiment.

[0149] 13(a) shows a three-dimensional point cloud 1430, FIG. 13(b) shows a model candidate 1460 for pipe A, and FIG. 13(c) shows a model candidate 1470 for pipe B.

[0150] A model candidate 1460 for the piping A includes a plurality of model shapes 1461, 1462, and 1463, each of which is different in shape, and a model candidate 1470 for the piping B includes a plurality of model shapes 1471, 1472, and 1473, each of which is different in shape.

[0151] The processing unit 53 compares the labeled point cloud 1450 (point cloud of black circles) with each of a plurality of model shapes 1461, 1462, and 1463 of the piping A, and replaces a specific area with the model shape that is the closest in shape. If there is no optimal model shape among the plurality of model shapes 1461, 1462, and 1463, the processing unit 53 may adjust the dimensions and shape of the model shape. Here, as one example, the model shape that is the closest in shape is the model shape that is the smallest in total distance between the model shape and each point of the point cloud.

[0152] In cases where there is a possibility that the texture or material may be different for the same shape, multiple textures and multiple materials may also be collated. For example, the color (RGB) of the point cloud is compared with the color of the texture included in the model shape information, and the closest color is set. Specifically, for each point of the point cloud, the difference between the texture color information (RGB) for the model at the closest location and the RGB value of the point cloud is calculated, and the sum of the absolute values ​​is calculated, but the comparison method is not important.

[0153] Furthermore, the processing unit 53 compares the labeled point cloud 1450 (point cloud of black circles) with each of the multiple model shapes 1471, 1472, and 1473 of the piping B, and replaces a specific area with the model shape that is the closest in shape. If there is no optimal model shape, the processing unit 53 may adjust the dimensions and shape of the model shape. If there is no optimal model shape among the multiple model shapes 1471, 1472, and 1473, the processing unit 53 may adjust the dimensions and shape of the model shape.

[0154] FIG. 14 is an explanatory diagram of the operation screen according to this embodiment.

[0155] FIG. 14 shows a display screen 1000 displayed on the display 308 of the terminal device 3 in step S10 of the sequence diagram shown in FIG.

[0156] The display control unit 34 of the terminal device 3 causes the display screen 1000 to display an operation screen 1500 including a generated shape setting screen 1510 and a generated shape display screen 1560, and a confirmation button 1300.

[0157] The generated shape setting screen 1510 is an example of a second reception screen for receiving a shape setting operation for setting one piece of three-dimensional shape information from a plurality of pieces of three-dimensional shape information, and includes generated shape setting boxes 1512 and 1514.

[0158] The generated shape setting box 1512 is used to set a three-dimensional shape generated based on the model shape information of the piping model A, and the generated shape setting box 1514 is used to set a three-dimensional shape generated based on the model shape information of the piping model B.

[0159] The display control unit 34 causes the generated shapes 1562 and 1564 to be displayed on the generated shape display screen 1560 in association with the names of the multiple pieces of model shape information.

[0160] Generated shape 1562 indicates a three-dimensional shape generated based on model shape information of piping model A, and generated shape 1564 indicates a three-dimensional shape generated based on model shape information of piping model B.

[0161] When the generated shape setting box 1512 or 1514 is pointed to with a pointing device such as the mouse 312, the reception unit 32 of the terminal device 3 receives the shape setting operation, and when the confirmation button 1300 is operated, confirms the shape setting operation.

[0162] Then, as described in step S11 of Fig. 6, the transmitting / receiving unit 31 transmits input information including the shape setting information by the shape setting operation received by the receiving unit 32 to the server 5. Note that in Fig. 14, either the generated shape 1562 or the generated shape 1564 is selected based on the user's operation, but the processing unit 53 of the server 5 may automatically select, from the generated shape 1562 or the generated shape 1564, the one closer to the labeled point cloud, regardless of the user's operation. Here, the closer generated shape is, for example, the generated shape with the smallest total distance calculated between the generated shape and each point of the point cloud.

[0163] FIG. 15A is a flowchart showing an example of the virtual space operation process according to the present embodiment, and shows the process of the server 5 corresponding to step S107 in the sequence diagram shown in FIG.

[0164] The generation unit 56 of the server 5 generates a virtual space screen of a viewpoint based on the operation information generated in step S103 and received in step S106 (step S41).

[0165] The determination unit 55 determines whether an article included in the virtual space screen is an article for which additional information is to be displayed based on the input information received in step S106 (step S42). Specifically, when an article is specified based on the input information, the determination unit 55 sets the article as a display target, and does not set unspecified articles as display targets. Furthermore, when no article is specified based on the input information, the determination unit 55 sets all articles as display targets.

[0166] Here, the input information includes at least one of a user ID that identifies the user and identification information that identifies the item, and when the user ID is received, the item corresponding to the user ID is identified as described in step S107.

[0167] This allows additional information to be displayed for items that are likely to be of interest to the user and that are identified based on the user ID and identification information, making it possible to appropriately display additional information according to the user's interests.

[0168] The generation unit 56 generates a virtual space screen on which dimension lines and dimensions are not superimposed and displayed for the articles determined not to be displayed in step S42 (step S43).

[0169] The determination unit 55 determines whether the object to be displayed protrudes from the virtual space screen (step S44). Specifically, as shown in FIG. 15B, when the protrusion amount or ratio of the object to be displayed from the virtual space screen is equal to or greater than a threshold, the determination unit 55 determines that the object protrudes.

[0170] 15B is a flow chart showing an example of the protrusion judgment process of the present embodiment. First, the judgment unit 55 judges whether or not it is unclear what angle of view the virtual space screen displayed on the display is actually displayed at, and if the angle of view is unclear (step S440: Yes), the process proceeds to step S441. On the other hand, if the angle of view is not unclear (step S440: No), the process proceeds to step S442.

[0171] If the angle of view is unknown, the determination unit 55 sets virtual viewpoint coordinates (coordinates of the position of the viewpoint from which the virtual space is viewed) and estimates the angle of view based on the virtual viewpoint coordinates (step S441). Specifically, it calculates the angle of view in the vertical and horizontal directions displayed on the display from the set virtual viewpoint coordinates. At this time, it is also possible to set multiple virtual viewpoint coordinates to estimate the angle of view and select an appropriate angle of view based on the estimation result.

[0172] Next, the determination unit 55 calculates virtual viewpoint coordinates based on a pre-given angle of view if the angle of view is not unknown, and based on the angle of view estimated in step S441 if the angle of view is unknown (step S442). Next, the determination unit 55 calculates a three-dimensional space corresponding to the virtual space screen based on the angle of view and the virtual viewpoint coordinates (step S443). Specifically, the determination unit 55 calculates a three-dimensional space formed by four half lines OA, OB, OC, and OD using the virtual viewpoint coordinates O and the four vertices A, B, C, and D of the virtual space screen.

[0173] Then, if the amount of the object protruding out of the three-dimensional space (protruding amount) is within the threshold (step S444: Yes), the judgment unit 55 judges that the object does not protrude out of the virtual space screen (step S445). On the other hand, if the protruding amount is not within the threshold (step S444: No), the judgment unit 55 judges that the object protrudes out of the virtual space screen (step S446). The protruding amount is, for example, the number of coordinates that are out of the three-dimensional space among the coordinate group representing the shape of the object. The coordinate group representing the shape of the object is the coordinates of its eight vertices when the object is a cube, and the coordinates of its six vertices when the object is a triangular prism. Furthermore, if the shape of the object is represented by a triangular mesh, the coordinate group representing the shape of the object is a set of coordinates of the vertices of each mesh. Furthermore, the coordinate group representing the shape of the object may be other than the coordinates of the vertices described above, as long as it is a set of coordinates that can represent the shape of the object.

[0174] The threshold value used in step S444 is, for example, a value previously set by an experiment or the like. For example, when the threshold value is 0, if even one of the coordinate groups representing the shape of the article is outside the three-dimensional space, the judgment unit 55 judges that the article is protruding from the screen. When the threshold value is 2, if three or more of the coordinate groups representing the shape of the article are outside the three-dimensional space, the judgment unit 55 judges that the article is protruding from the screen.

[0175] The generation unit 56 generates a virtual space screen on which dimension lines and dimensions are not displayed in a superimposed manner for the object determined to be protruding in step S44 (step S43).

[0176] The determination unit 55 determines whether the visible area of ​​the face of the article to be displayed exceeds a threshold value (step S45).

[0177] The generation unit 56 generates a virtual space screen on which dimension lines and dimensions are not superimposed and displayed for an article whose visible area is determined to be equal to or smaller than the threshold in step S45 (step S43).

[0178] The processing unit 53 calculates the distance from the position of the virtual camera to the article to be displayed (step S46), and the determination unit 55 determines whether the distance to the article to be displayed exceeds a threshold value (step S47).

[0179] When the virtual space screen is a two-dimensional image, the processing unit 53 performs image analysis on the two-dimensional image to calculate the distance from the shooting position to the item to be displayed.

[0180] The generation unit 56 generates a virtual space screen on which dimension lines and dimensions are not superimposed and displayed for the article whose distance is determined to be equal to or less than the threshold in step S47 (step S43).

[0181] The judgment unit 55 judges whether the item to be displayed is placed at the center of the virtual space screen (step S48), and if it is placed at the center, the generation unit 56 generates a virtual space screen that superimposes and displays all dimension lines and dimensions stored in the virtual space management DB 5006 in association with the item (step S49).

[0182] On the other hand, if the object is not positioned in the center, the generation unit 56 generates a virtual space screen that superimposes and displays some of the dimension lines and dimensions stored in the virtual space management DB 5006 in association with the object (step S50).

[0183] Here, in step S50, the generation unit 56 may generate a virtual space screen on which dimension lines and dimensions are not displayed in a superimposed manner.

[0184] If at least one of the coordinates representing the shape of the article is outside the three-dimensional space in step S44, the generation unit 56 can hide all additional information related to the article. In this case, if a part of the article (such as two points constituting a side of the shape of the article or four points constituting a face) is within the three-dimensional space, the generation unit 56 may display additional information related to that part.

[0185] Alternatively, the determination unit 55 may make a determination with respect to two threshold levels in steps S44, S45, and S47, respectively, and the generation unit 56 may generate a virtual space in any one of steps S43, S49, and S50.

[0186] 15A, if at least one of the position, orientation, and angle of view of the virtual camera is changed by the user's input operation, the process returns to step S41. At this time, the generation unit 56 may generate a virtual space screen according to the changed information, and the determination unit 55 may make the determinations of steps S42, S44, S45, S47, and S48 according to the changed information, and execute the process of step S43, S49, or S50.

[0187] That is, when an object for which additional information is displayed extends beyond the virtual space screen by changing at least one of the position, direction, and angle of view of the virtual camera, the generation unit 56 displays additional information whose dimensions, dimension lines, etc. have been changed according to the changed information. The generation unit 56 can generate, for example, a virtual space screen that displays additional information (changed additional information) that is different in at least one of the number and type of information from the additional information displayed before the change (pre-change additional information), or a virtual space screen that displays neither the pre-change additional information nor the post-change additional information. In addition, when an object for which additional information is not displayed because it extends beyond the virtual space screen no longer extends beyond the virtual space screen after the change, the generation unit 56 can generate a virtual space screen that displays additional information related to the object.

[0188] FIG. 16 is an explanatory diagram of a three-dimensional image according to this embodiment.

[0189] 16(a) is a diagram showing the positions of a virtual camera and a predetermined area when a celestial sphere image is a three-dimensional sphere. The virtual camera IC corresponds to the position of the viewpoint of a user viewing a celestial sphere image CE displayed as a three-dimensional sphere.

[0190] FIG. 16(b) is a three-dimensional perspective view of FIG. 16(a), and FIG. 16(c) is a diagram showing a predetermined area image when displayed on a display.

[0191] Fig. 16(b) represents the celestial sphere image CE shown in Fig. 23 as a three-dimensional stereoscopic sphere CS. If the celestial sphere image CE generated in this manner is a stereoscopic sphere CS, then the virtual camera IC is located inside the celestial sphere image CE as shown in Fig. 23.

[0192] The predetermined area T in the omnidirectional image CE is an imaging area of ​​the virtual camera IC, and is specified by predetermined area information indicating the imaging direction and angle of view of the virtual camera IC in a three-dimensional virtual space including the omnidirectional image CE.

[0193] Also, zooming of the predetermined area T can be expressed by moving the virtual camera IC closer to or farther away from the omnidirectional image CE. The predetermined area image Q is an image of the predetermined area T in the omnidirectional image CE. Therefore, the predetermined area T can be specified by the angle of view α and the distance f from the virtual camera IC to the omnidirectional image CE.

[0194] That is, the display control unit 34 changes the display area of ​​the three-dimensional image 3G displayed on the display units 20, 520 by changing the position and orientation of the virtual camera IC located at the viewpoint position for viewing the three-dimensional image 3G.

[0195] Although the above description has been given of an example in which a celestial sphere image is used as the three-dimensional image displayed by the display unit, the same applies when three-dimensional point cloud data is used. A three-dimensional point cloud is arranged in a virtual space, and a virtual camera is arranged in the virtual space. A three-dimensional image is obtained by projecting the three-dimensional point cloud onto a predetermined projection surface in the virtual space based on the predetermined area information indicating the viewpoint position, shooting direction, and angle of view of the virtual camera. In addition, the display area of ​​the three-dimensional image is changed by changing the viewpoint position and direction of the virtual camera.

[0196] FIG. 17 is an explanatory diagram of a virtual space screen according to this embodiment.

[0197] FIG. 17 shows a display screen 1000 displayed on the display 308 of the terminal device 3 in step S109 of the sequence diagram shown in FIG.

[0198] The display control unit 34 of the terminal device 3 causes the display screen 1000 to display a virtual space screen 1600 seen from the virtual camera IC described in Figure 16, which includes a beam 1611, pillars 1613, 1615, and walls 1620 and 1630, which are examples of objects having a three-dimensional shape.

[0199] The display control unit 34 superimposes and displays the dimension lines 1711A and 1711B, the dimensions 1811A and 1811B, the target surface 1711C, and the dimension (area) 1811C on the beam 1611.

[0200] The display control unit 34 displays dimension lines 1713A, 1713B and dimensions 1813A, 1813B superimposed on the column 1613, and displays dimension lines 1715A, 1715B and dimensions 1815A, 1815B superimposed on the column 1615.

[0201] The display control unit 34 does not superimpose and display dimension lines and dimensions for the walls 1620 and 1630.

[0202] In other words, the generation unit 56 which generates the display screen 1000 displays all of the dimension lines and dimensions for the beam 1611, but does not display some of the dimension lines and dimensions for the wall 1630 which is at a different distance from the viewpoint than the beam 1611 and is at a longer distance from the viewpoint than the beam 1611.

[0203] This allows the user to grasp different additional information for each target part that is at a different distance from the viewpoint.

[0204] The generating unit 56 may display dimension lines and part of the dimensions for the wall 1630.

[0205] FIG. 18 is a second explanatory diagram of the virtual space screen according to this embodiment.

[0206] The display control unit 34 of the terminal device 3 causes the display screen 1000 to display a virtual space screen 1600 including a beam 1611, pillars 1613 and 1615, and walls 1620 and 1630, similar to FIG.

[0207] In the display screen 1000 shown in FIG. 18, unlike the display screen 1000 shown in FIG. 17, the target surface 1711C of the pillar 1613 is not visible, and the dimension line 1713B and the dimension 1813B corresponding to the width of the target surface 1711C are not displayed.

[0208] That is, the pillar 1613 includes a target surface 1613C, and the generation unit 56 displays additional information on the target surface 1613C based on the size of the target surface 1613C in the virtual space screen 1600 if the target surface 1613C is equal to or larger than a threshold value.

[0209] In this way, when the target surface 1613C is large enough to be visible, the additional information is displayed, and when the target surface 1613C is large enough to be difficult to see, the additional information is not displayed, so that the user can properly understand which target surface the additional information corresponds to.

[0210] Furthermore, in the display screen 1000 shown in FIG. 18, unlike the display screen 1000 shown in FIG. 17, the beam 1611 extends beyond the virtual space screen 1600, and the dimension lines 1711A, 1711B, dimensions 1811A, 1811B, target surface 1711C, and dimension (area) 1811C for the beam 1611 are all not displayed.

[0211] That is, the generation unit 56 displays additional information for an object when the amount or rate of protrusion of the object from the virtual space screen 1600 is less than a threshold value.

[0212] As a result, by not displaying additional information for an item that protrudes from the virtual space screen 1600 and is unlikely to interest the user, additional information can be appropriately displayed according to the user's interests.

[0213] FIG. 19 is a third explanatory diagram of the virtual space screen according to this embodiment.

[0214] The display control unit 34 of the terminal device 3 causes the display screen 1000 to display a virtual space screen 1600 including walls 1620 and 1630 .

[0215] In the display screen 1000 shown in FIG. 19, unlike the display screen 1000 shown in FIG. 17, the display control unit 34 superimposes and displays dimension lines 1720A, 1720B, 1720C, 1720D, dimensions 1820A, 1820B, 1820C, 1820D, and dimension (area) 1820C on the wall 1620.

[0216] Similarly, the display control unit 34 displays dimension lines 1730A and 1730B and dimensions 1830A and 1830B corresponding to the opening 1635 on the wall 1630 in a superimposed manner.

[0217] That is, the generation unit 56 displays all dimension lines and dimensions for the wall 1630 when the distance from the viewpoint position to the wall 1630 is a first distance that is shorter than a threshold, as in FIG. 19, and does not display the dimension lines and dimensions for the wall 1630 when the distance from the viewpoint position to the wall 1630 is a second distance that is longer than a threshold different from the first distance, as in FIG. 17.

[0218] This makes it possible to display different additional information when the distance from the viewpoint to the target portion changes.

[0219] Note that the generating unit 56 may display a dimension line and part of the dimension for the wall 1630 when the distance from the viewpoint position to the wall 1630 is a second distance.

[0220] FIG. 20 is a fourth explanatory diagram of the virtual space screen according to this embodiment.

[0221] The display control unit 34 of the terminal device 3 causes the virtual space screen 1600 including the wall 1630 to be displayed on the display screen 1000 .

[0222] In the display screen 1000 shown in FIG. 20, unlike the display screen 1000 shown in FIG. 19, the wall 1630 does not extend beyond the virtual space screen 1600, and the display control unit 34 further superimposes and displays dimension lines 1730C, 1730D, and 1730E and dimensions 1830C, 1830D, and 1830E on the wall 1630.

[0223] FIG. 21 is a fifth explanatory diagram of the virtual space screen according to this embodiment.

[0224] The display control unit 34 of the terminal device 3 causes a virtual space screen 1600 including walls 1620 and 1630 to be displayed on the display screen 1000 .

[0225] That is, for the wall 1630 that is located at the center, the generation unit 56 displays dimension lines 1730A, 1730B and dimensions 1830A, 1830B corresponding to the opening 1635 in a superimposed manner, but for the wall 1620 that is not located at the center, the generation unit 56 does not display dimension lines and dimensions in a superimposed manner.

[0226] FIG. 22 is a fifth explanatory diagram of the virtual space screen according to this embodiment.

[0227] The display control unit 34 of the terminal device 3 causes a virtual space screen 1600 including walls 1620 and 1630 to be displayed on the display screen 1000 .

[0228] That is, for wall 1620 that is located in the center, generation unit 56 displays dimension lines 1720B, 1720C, 1720D, dimensions 1820B, 1820C, 1820D, and dimension (area) 1820C in a superimposed manner, but for wall 1630 that is not located in the center, generation unit 56 does not display dimension lines and dimensions in a superimposed manner.

[0229] FIG. 23 is a sixth explanatory diagram of the virtual space screen according to the present embodiment.

[0230] The display control unit 34 of the terminal device 3 causes the display screen 1000 to display a virtual space screen 1600 including a beam 1611, pillars 1613, 1615, walls 1620 and 1630, as in FIG.

[0231] Pop-up screens 1910A to 1910D are other examples of additional information for an article, and show images of beam 1611 as viewed from above, left, right, and below, respectively.

[0232] FIG. 24 is a seventh explanatory diagram of the virtual space screen according to this embodiment.

[0233] The display control unit 34 of the terminal device 3 causes the virtual space screen 1600 including the wall 1640 to be displayed on the display screen 1000 .

[0234] The display control unit 34 first displays the dimension line 1740 and the dimension 1840 in a superimposed manner on the wall 1640 .

[0235] In this state, when the user performs an operation to select dimension line 1740 or dimension 1840 through reception unit 32, display control unit 34 further superimposes and displays dimension lines 1740a, 1740b, dimensions 1840a, 1840b, dimension (area) 1840c, and comment 1940 on wall 1640. Comment 1940 is another example of additional information for the article.

[0236] The display control unit 34 can display, for example, a comment on the dimension 1840 in the comment 1940. The comment on the dimension 1840 is, for example, a comment on the circumstances when the dimension of the wall 1640 was set, the merits of the set dimension, and the like. The comment (an example of additional information) for each target part can be created in advance and stored in the model shape management DB 5005 or the like. In this case, the storage / reading unit 59 reads out the comment for the target part selected by the user, and the display control unit 34 displays the read comment by superimposing it on the target part.

[0237] The processing unit 53 may generate a comment for the target part selected by the user using a multimodal AI (Artificial Intelligence). For example, a combination of data of each target part and text related to each target part can be learned by the multimodal AI by contrastive learning known as CLIP (Contrastive Language Image Pre-training). Here, the data of each target part is three-dimensional point cloud information or three-dimensional shape information corresponding to the target part. Moreover, the text related to each target part is text created by a designer or an expert, etc., indicating the design intention or know-how of the target part. In this case, the processing unit 53 generates a comment for the target part selected by the user using the learned multimodal AI, and the display control unit 34 displays the generated comment by superimposing it on the target part.

[0238] FIG. 25 is an explanatory diagram of input information according to this embodiment.

[0239] The display control unit 34 of the terminal device 3 causes a dimension type selection screen 1920 and an item selection screen 1930 to be displayed on the display screen 1000 .

[0240] The generating unit 56 displays the dimension lines and dimensions of the type specified on the dimension type selection screen 1920 accepted by the accepting unit 32 for the item specified on the item selection screen 1930 accepted by the accepting unit 32. This makes it possible to appropriately display additional information according to the user's interest.

[0241] The generating unit 56 may generate a virtual space screen that displays specified additional information described below instead of or in addition to the above-mentioned additional information. The additional information is displayed by changing at least one of the position, orientation, and angle of view of the virtual camera at the virtual viewpoint coordinates, whereas the specified additional information is displayed based on the specified information received by the receiving unit 32. The specified information is input by, for example, the user specifying a position in the virtual space screen or the displayed additional information. The specified additional information is, for example, specified dimension line position information (specified dimension line position information) or specified dimension information (dimension information corresponding to the specified dimension line position information, or specified dimension information). The specified dimension line position information and the specified dimension information are examples of specified additional information.

[0242] The designation operation is, for example, an operation of designating the position of a three-dimensional shape included in the virtual space screen. The processing unit 53 can determine designated dimension line position information based on the designated position.

[0243] The designation operation may be an operation of designating dimension line position information or dimension information on a virtual space screen on which the dimension line position information and the dimensional information are superimposed. The processing unit 53 can treat the designated dimension line position information or dimensional information as designated dimension line position information or designated dimensional information.

[0244] The additional information and the designated additional information may be displayed in different formats. For example, the additional information and the designated additional information may be displayed in different colors, or in different sizes, so that it is easy to visually recognize which additional information is the designated additional information.

[0245] When a part of a target portion protrudes from the virtual space screen, the generation unit 56 may generate a virtual space screen that does not display additional information for the target portion, but displays specified additional information for the target portion.

[0246] FIG. 26 is an eighth explanatory diagram of the virtual space screen according to this embodiment. FIG. 26 shows an example in which the designated additional information is displayed on the virtual space screen of FIG. 18. The difference from FIG. 18 is that the position information of the dimension line 1711A is specified as the designated dimension line position information, and the dimension 1811A is specified as the designated dimension information. Since a part of the beam 1611 protrudes from the virtual space screen 1600, the dimension line 1711A and the dimension 1811A for the beam 1611 are not displayed in FIG. 18, but the designated dimension line 1711A' and the designated dimension 1811A' for the beam 1611 are displayed in FIG. 26. Note that the designated dimension line 1711A' and the designated dimension 1811A' are displayed in the same format as the dimension line 1713A and the dimension 1813A in FIG. 26, but they may be displayed in different formats.

[0247] 27 is a flowchart showing an example of a virtual space screen generation process when the specified additional information is included. First, the determination unit 55 determines whether or not the additional information corresponding to the item is specified by the user (step S500). If the additional information is specified by the user (step S500: Yes), the process proceeds to step S501. On the other hand, if the additional information is not specified by the user (step S500: No), the process proceeds to step S502.

[0248] In step S501, the generation unit 56 generates a virtual space screen that displays additional information (specified additional information) corresponding to an item even if a part of the item protrudes from the virtual space screen.

[0249] In step S502, if a part of an item protrudes from the virtual space screen, the generation unit 56 generates a virtual space screen that does not display additional information corresponding to that item.

[0250] In this way, the designated additional information is displayed even if a part of the target part is outside the virtual space screen. This allows the user to designate additional information that is of high interest to the user, and to visually recognize the additional information of high interest even if the viewpoint or angle of view of the virtual space screen is changed and a part of the object is outside the virtual space screen.

[0251] ●Summary● [First aspect] As described above, the server 5 or the terminal device 3 integrated with the server 5, which is an example of an information processing device according to an embodiment of the present invention, displays a virtual space screen 1600 showing a space, and includes a generation unit 56 that generates a display screen 1000 that displays additional information about an object, which is generated based on the distance from the position of the viewpoint viewing the virtual space screen 1600 to the object in the virtual space screen 1600, superimposed on the virtual space screen 1600. The virtual space screen 1600 is an example of a space image, the generation unit 56 is an example of a display screen generation means, and the object is an example of a target portion.

[0252] This allows the user to appropriately grasp additional information about an item according to the distance from the viewpoint to the item.

[0253] In other words, an object that is close to the viewpoint has a large image area on the virtual space screen 1600 and is easy to see even when additional information is displayed, but an object that is far from the viewpoint has a small image area on the virtual space screen 1600 and is difficult to see when additional information is displayed.

[0254] Therefore, by displaying a large amount of additional information for items that are close to the viewpoint position and are likely to interest the user, and displaying less additional information or no additional information for items that are far from the viewpoint position and are unlikely to interest the user, the additional information can be displayed sufficiently and unclutteredly according to the user's interests.

[0255] [Second aspect] In the first embodiment, the additional information includes dimension lines and dimensions for the article, which allows the user to properly understand the dimension lines and dimensions displayed on the display screen 1000.

[0256] [Third aspect] In the first or second aspect, the virtual space screen 1600 is a three-dimensional image, and the generation unit 56 changes the display area of ​​the three-dimensional image displayed on the display screen 1000 by changing at least one of the position, orientation, and angle of view of a virtual camera located at the viewpoint position.

[0257] In this case, the processing unit 53 calculates the distance from the position of the virtual camera to the item to be displayed, but if the virtual space screen is a two-dimensional image, the processing unit 53 calculates the distance from the shooting position to the item to be displayed by performing image analysis of the two-dimensional image.

[0258] This allows the user to easily grasp additional information about the item in the three-dimensional image.

[0259] [Fourth aspect] In the third aspect, the generation unit 56 generates a display screen 1000 that displays all of the dimension lines and dimensions, which are an example of first additional information, for the wall 1630, when the distance from the viewpoint position to the wall 1630, which is an example of an object, is a first distance that is shorter than a threshold value, and generates a display screen 1000 that displays some of the dimension lines and dimensions, which are an example of second additional information that differs from the first additional information in at least one of the number and type of information, for the wall 1630, or does not display the dimension lines and dimensions, when the distance from the viewpoint position to the wall 1630 is a second distance that is longer than a threshold value different from the first distance.

[0260] This makes it possible to display different additional information when the distance from the viewpoint to the target portion changes.

[0261] [Fifth aspect] In the third mode, the additional information is displayed by changing at least one of the position, orientation, and angle of view of the virtual camera, and the generation unit 56 generates a display screen 1000 that superimposes the additional information and specified additional information that is displayed based on the specified information accepted by the input accepting means on the virtual space screen 1600.

[0262] This allows the designated additional information to be displayed by a designated operation.

[0263] [Sixth aspect] In any of the first to fifth aspects, the generation unit 56 generates a display screen 1000 that displays all of the dimension lines and dimensions, which are an example of first additional information, for a beam 1611, which is an example of a first object, and generates a display screen 1000 that displays some of the dimension lines and dimensions, which are an example of second additional information that differs from the first additional information in at least one of the number and type of information, or does not display the dimension lines and dimensions, for a wall 1630, which is an example of a second object that is at a different distance from the viewpoint than the beam 1611 and is at a longer distance from the viewpoint than the beam 1611.

[0264] This allows the user to grasp different additional information for each target part that is at a different distance from the viewpoint.

[0265] [Seventh aspect] In any of the first to sixth aspects, a pillar 1613, which is an example of an object, includes a target surface 1613C, and the generation unit 56 generates a display screen 1000 that displays additional information for the target surface 1613C based on the size of the target surface 1613C in the virtual space screen 1600 if the target surface 1613C is equal to or greater than a threshold, and generates a display screen 1000 that does not display additional information for the target surface 1613C if the target surface 1613C is less than the threshold.

[0266] In this way, when the target surface 1613C is large enough to be visible, the additional information is displayed, and when the target surface 1613C is large enough to be difficult to see, the additional information is not displayed, so that the user can properly understand which target surface the additional information corresponds to.

[0267] [Eighth aspect] In the first to seventh embodiments, the generation unit 56 generates a display screen 1000 that displays additional information for an item based on the amount or percentage of the item sticking out from the virtual space screen 1600, and generates a display screen 1000 that does not display additional information for an item if the amount or percentage of the item sticking out from the virtual space screen 1600 is less than a threshold value, and generates a display screen 1000 that does not display additional information for an item if the amount or percentage of the item sticking out is equal to or greater than a threshold value.

[0268] As a result, by not displaying additional information for an item that protrudes from the virtual space screen 1600 and is unlikely to interest the user, additional information can be appropriately displayed according to the user's interests.

[0269] [Ninth aspect] In the first to eighth embodiments, when the target portion on which the third additional information is displayed extends beyond the virtual space screen 1600, the generation unit 56 displays fourth additional information that differs from the third additional information in at least one of the number and type of information, or generates a display screen 1000 that displays neither the third additional information nor the fourth additional information.

[0270] This makes it possible to display additional information in which the dimensions, the number of dimension lines, etc. displayed have been changed in accordance with the change in the virtual space screen 1600.

[0271] [Tenth aspect] In the first to ninth embodiments, the generation unit 56 generates a display screen 1000 that displays additional information for an item specified based on input information received by the reception unit 32, which is an example of an input reception means.

[0272] This makes it possible to appropriately display additional information according to the user's interests by displaying additional information for an item that is identified based on the input information and that is likely to be of interest to the user.

[0273] [Eleventh aspect] In a tenth aspect, the input information includes at least one of a user ID, which is an example of user information for identifying a user, and specific information for identifying an item. This allows additional information to be appropriately displayed according to the user's interests.

[0274] [12th aspect] In the first to eleventh embodiments, the generating unit 56 generates a display screen 1000 that displays additional information, the type, number, etc. of which have been specified based on the input information received by the receiving unit 32, for the article.

[0275] This makes it possible to appropriately display additional information according to the user's interests.

[0276] [13th aspect] The terminal device 3, which is an example of an information processing device according to an embodiment of the present invention, is equipped with a display control unit 34 that displays a virtual space screen 1600 showing a space, and causes a display screen 1000 to be displayed on a display 308. The display screen 1000 displays additional information about an object, which is generated based on the distance from the viewpoint for viewing the virtual space screen 1600 to the object in the virtual space screen 1600, superimposed on the virtual space screen 1600.

[0277] [14th aspect] An information processing method executed by a server 5, which is an example of an information processing device according to one embodiment of the present invention, or a terminal device 3 integrated with the server 5, executes a generation step S107 of generating a display screen 1000 that displays a virtual space screen 1600 showing a space, and superimposes additional information about an object on the virtual space screen 1600, the additional information being generated based on the distance from the viewpoint position for viewing the virtual space screen 1600 to the object in the virtual space screen 1600.

[0278] [15th aspect] A program according to one embodiment of the present invention causes a computer to execute a generation step of generating a display screen 1000 that displays a virtual space screen 1600 showing a space, and superimposes additional information about an object on the virtual space screen 1600, the additional information being generated based on the distance from the viewpoint from which the virtual space screen 1600 is viewed to the object in the virtual space screen 1600.

[0279] [16th aspect] A three-dimensional shape processing system 1, which is an example of an information processing system according to one embodiment of the present invention, includes a terminal device 3 and a server 5 connected to the terminal device 3 via a communication network 100. The server 5 displays a virtual space screen 1600 showing a space, and includes a generation unit 56 that generates a display screen 1000 that superimposes additional information about an object in the virtual space screen 1600, which is generated based on the distance from the viewpoint position for viewing the virtual space screen 1600 to the object, on the virtual space screen 1600, and a transmission / reception unit 51 that transmits display screen 1000 information showing the display screen 1000 to the terminal device 3. The terminal device 3 includes a transmission / reception unit 31 that receives the display screen 1000 information transmitted from the server 5, and a display control unit 34 that causes the display screen 1000 to be displayed on the display unit. [Explanation of symbols]

[0280] 1. 3D shape processing system 100 Communication Network 3. Terminal device (an example of an input / output device) 31 Transmitting / receiving unit (an example of a transmitting means or a receiving means) 32 Reception unit (an example of a reception means) 34 Display control unit (an example of a display control means) 36 Judgment Department 39 Memory / read unit (an example of a memory control means) 308 Display (an example of a display unit) 3000 storage section 5 Server (an example of a 3D shape generation device) 51 Transmitting / receiving unit (an example of a transmitting means or a receiving means) 53 Processing unit (an example of a three-dimensional information generating means) 55 Judgment Department 56 Generation unit (an example of a display screen generation means) 57 Setting section 59 Memory / read unit (an example of a memory control means) 5000 storage section 5001 Setting information management DB (an example of setting information management means) 5002 Memory processing management DB (an example of memory processing management means) 5003 Point cloud management DB (an example of point cloud management means) 5004 Three-dimensional shape management DB (an example of a three-dimensional shape management means) 5005 Model shape management DB 5006 Virtual space management database 1000 display screen 1100 User information display screen 1200 Setting screen 1210 Point Cloud Setting Screen 1212, 1214 Point Cloud Settings Box 1220 Processing setting screen 1222, 1224, 1226 Processing setting box 1230 Model information setting screen (example of reception screen) 1231, 1234, 1235 Model Information Setting Box 1232, 1233 Model candidate setting box 1240 Model Display Screen 1242, 1243 model candidates 1244 Model Shape 1300 Confirm button (example of instruction reception screen) 1410 First 3D point cloud 1420 Second 3D Point Cloud 1430 3D point cloud 1440 Unwanted points 1450 Labeled Point Cloud 1460, 1470 model candidates 1461, 1462, 1463, 1471, 1472, 1473 Model Shape 1500 operation screen 1510 Generation shape setting screen (an example of the second reception screen) 1512, 1514 Generated shape setting box 1560 Generated shape display screen 1562, 1564 Generated shape 1600 Virtual space screen (example of a space image) 1611 Beam (an example of a target part or item) 1613, 1615 Pillars (examples of target parts and items) 1620, 1630, 1640 Wall (example of target part, item) 1711~1740 Dimension lines (example of additional information) 1711A' Designated dimension line (example of designated additional information) 1811~1840 Dimensions (example of additional information) 1811A' Specified dimensions (example of specified additional information)

Claims

1. An information processing device comprising: a display screen generating means for generating a display screen that displays a spatial image showing a space, and superimposes additional information about the target part in the spatial image, the additional information being generated based on the distance from the viewpoint position for viewing the spatial image to the target part in the spatial image.

2. The information processing apparatus according to claim 1 , wherein the additional information includes dimension lines and dimensions for the target portion.

3. the aerial image is a three-dimensional image; 2 . The information processing apparatus according to claim 1 , wherein the display screen generating means changes at least one of the position, the orientation, and the angle of view of a virtual camera located at the viewpoint position, thereby changing the display area of ​​the three-dimensional image to be displayed on the display screen.

4. The display screen generating means generating the display screen for displaying first additional information with respect to the target portion when a distance from the viewpoint position to the target portion is a first distance; 4. An information processing device as described in claim 3, wherein when the distance from the viewpoint position to the target portion is a second distance different from the first distance, second additional information is displayed for the target portion, the second additional information differing in at least one of the number and type of information from the first additional information, or the display screen is generated so that neither the first additional information nor the second additional information is displayed.

5. the additional information is displayed by changing at least one of a position, a direction, and an angle of view of the virtual camera; 4. The information processing apparatus according to claim 3, wherein the display screen generating means generates a display screen for displaying the additional information and designated additional information displayed based on designated information received by an input receiving means, superimposed on the spatial image.

6. The display screen generating means generating the display screen for displaying first additional information for a first target portion; 2. The information processing device according to claim 1, wherein the display screen is generated by displaying second additional information, which differs from the first additional information in at least one of the number and type of information, for a second target portion that is a different distance from the viewpoint position to the first target portion, or by displaying neither the first additional information nor the second additional information.

7. the target portion includes a target surface; The display screen generating means The information processing apparatus according to claim 1 , wherein the display screen is generated so as to display or not display the additional information with respect to the target surface based on the size of the target surface in the spatial image.

8. The display screen generating means The information processing apparatus according to claim 1 , wherein the display screen is generated so as to display or not display the additional information for the target portion based on the extent to which the target portion extends beyond the spatial image.

9. 2. The information processing device according to claim 1, wherein the display screen generating means generates the display screen in which, when the target portion on which the third additional information is displayed extends beyond the spatial image, the display screen displays fourth additional information that differs from the third additional information in at least one of the number and type of information, or the display screen in which neither the third additional information nor the fourth additional information is displayed.

10. The display screen generating means 2 . The information processing apparatus according to claim 1 , wherein the display screen is generated to display the additional information for the target portion specified based on input information received by an input receiving unit.

11. The information processing apparatus according to claim 10 , wherein the input information includes at least one of user information for identifying a user and identification information for identifying the target portion.

12. The display screen generating means 2 . The information processing apparatus according to claim 1 , further comprising: a display screen for displaying, on the target portion, the additional information specified based on input information received by an input receiving unit.

13. An information processing device having a display control means for displaying a spatial image showing a space, and displaying on a display unit a display screen which displays additional information about the target part in the spatial image, which is generated based on the distance from the viewpoint position for viewing the spatial image to the target part in the spatial image, superimposed on the spatial image.

14. An information processing method executed by an information processing device, An information processing method that executes a generation step of generating a display screen that displays a spatial image showing a space, and superimposes additional information about the target part in the spatial image, which is generated based on the distance from the viewpoint position for viewing the spatial image to the target part in the spatial image.

15. A program that causes a computer to execute a generation step of generating a display screen that displays a spatial image showing a space and superimposes additional information about the target part on the spatial image, the additional information being generated based on the distance from the viewpoint position for viewing the spatial image to the target part in the spatial image.

16. An information processing system including a display device and an information processing device connected to the display device via a network, The information processing device includes: a display screen generating means for generating a display screen for displaying a spatial image showing a space, and displaying additional information for the target portion, which is generated based on a distance from a viewpoint position for viewing the spatial image to the target portion in the spatial image, superimposed on the spatial image; a transmitting means for transmitting display screen information indicating the display screen to the display device; Equipped with The display device includes: a receiving means for receiving the display screen information transmitted from the information processing device; A display control means for displaying the display screen on a display unit; An information processing system comprising:

Images