Information processing apparatus, information processing system, information processing method, and recording medium

Abstract

An information processing apparatus includes: display screen generation means configured to generate a display screen including a spatial image and additional information for a target portion, the spatial image being an image indicating a space including the target portion, the additional information being superimposed on the spatial image. The additional information is generated based on a distance from a position of a point of view from which the spatial image is viewed to the target portion in the spatial image.

Description

[DESCRIPTION] [Title of Invention] INFORMATION PROCESSING APPARATUS, INFORMATION PROCESSING SYSTEM, INFORMATION PROCESSING METHOD, AND RECORDING MEDIUM [Technical Field]

[0001] The present disclosure relates to an information processing apparatus, an information processing system, an information processing method, and a recording medium. [Background Art]

[0002] Patent Literature (PTL) 1 discloses an information processing apparatus including an acquisition unit, a specifying unit, a calculation unit, and a display control unit. The acquisition unit acquires a captured image generated by a terminal capturing an image of a target range, and point cloud data indicating a three-dimensional position of an object included in the target range measured by the terminal. The specifying unit specifies areas corresponding to a plurality of reinforcing bars in the target range, based on the point cloud data. The calculation unit calculates an interval between two adjacent reinforcing bars among the plurality of reinforcing bars, based on the point cloud data. The display control unit controls a display device to display the areas corresponding to the plurality of reinforcing bars and information indicating the interval such that the areas corresponding to the plurality of reinforcing bars are superimposed on the captured image.[Citation List][Patent Literature]

[0003] [PTL 1]Japanese Patent No. 7150290[Summary of Invention] [Technical Problem]

[0004] In the related art, it is difficult for a user to appropriately recognize additional information for a target portion in a spatial image.[Solution to Problem]

[0005] According to an embodiment of the present disclosure, an information processing apparatus includes display screen generation means. The display screen generation means generates a display screen including a spatial image and additional information for a target portion. The spatial image is an image indicating a space including the target portion. The additional information is superimposed on the spatial image. The additional information is generatedbased on a distance from a position of a point of view from which the spatial image is viewed to the target portion in the spatial image.According to an embodiment of the present disclosure, an information processing system includes the above-described information processing apparatus and a display apparatus connected to the information processing apparatus via a network. The information processing apparatus further includes transmitting means. The transmitting means transmits display screen information indicating the display screen to the display apparatus. The display apparatus includes receiving means and display control means. The receiving means receives the display screen information transmitted from the information processing apparatus. The display control means controls a display unit to display the display screen.According to an embodiment of the present disclosure, an information processing method includes generating a display screen including a spatial image and additional information for a target portion, the spatial image being an image indicating a space including the target portion, the additional information being superimposed on the spatial image. The additional information is generated based on a distance from a position of a point of view from which the spatial image is viewed to the target portion in the spatial image.According to an embodiment of the present disclosure, a recording medium carries computer readable code for controlling a computer system to carry out an information processing method. The information processing method includes generating a display screen including a spatial image and additional information for a target portion, the spatial image being an image indicating a space including the target portion, the additional information being superimposed on the spatial image. The additional information is generated based on a distance from a position of a point of view from which the spatial image is viewed to the target portion in the spatial image.[Advantageous Effects of Invention]

[0006] The configuration described above allows a user to appropriately recognize additional information for a target portion in a spatial image.[Brief Description of Drawings]

[0007] A more complete appreciation of embodiments of the present disclosure and many of the attendant advantages and features thereof can be readily obtained and understood from the following detailed description with reference to the accompanying drawings.[FIG. 1]FIG. l is a general arrangement diagram of a three-dimensional shape generation system according to an embodiment of the present disclosure.[FIG. 2]FIG. 2 is a hardware configuration diagram of a terminal apparatus and a server according to the embodiment.[FIG. 3]FIG. 3 is a functional block diagram of the three-dimensional shape generation system according to the embodiment.[FIG. 4]FIG. 4 is an illustration of an example of a setting information management table.[FIG. 5]FIG. 5 is an illustration of an example of a virtual space management table.[FIG. 6]FIG. 6 is a sequence diagram illustrating an example of a three-dimensional shape generation process.[FIG. 7]FIG. 7 is a sequence diagram illustrating an example of a virtual space operation process.[FIG. 8]FIG. 8 is an illustration of a display screen according to the embodiment.[FIG. 9]FIG. 9 is a flowchart illustrating an example of a three-dimensional shape generation process.[FIG. 10 A]FIG. 10A is an illustration of a registration process according to the embodiment.[FIG. 10B]FIG. 10B is an illustration of the registration process according to the embodiment.[FIG. 11]FIG. 11 is an illustration of a noise removal process according to the embodiment.[FIG. 12]FIG. 12 is an illustration of a segmentation process according to the embodiment.[FIG. 13 A]FIG. 13 A is an illustration of a model collation and replacement process according to the embodiment.[FIG. 13B]FIG. 13B is an illustration of a model collation and replacement process according to the embodiment.[FIG. 13C]FIG. 13C is an illustration of a model collation and replacement process according to the embodiment.[FIG. 14]FIG. 14 is an illustration of an operation screen according to the embodiment.[FIG. 15 A]FIG. 15A is a flowchart illustrating an example of a virtual space operation process.[FIG. 15B]FIG. 15B is a flowchart illustrating an example of a process for determining whether an article extends off-screen.[FIG. 16 A]FIG. 16A is an illustration of a three-dimensional image according to the embodiment.[FIG. 16B]FIG. 16B is an illustration of a three-dimensional image according to the embodiment.[FIG. 16C]FIG. 16C is an illustration of a three-dimensional image according to the embodiment.[FIG. 17]FIG. 17 is a diagram illustrating a virtual space screen according to the embodiment.[FIG. 18]FIG. 18 is a diagram illustrating a second example of the virtual space screen according to the embodiment.[FIG. 19]FIG. 19 is a diagram illustrating a third example of the virtual space screen according to the embodiment.[FIG. 20]FIG. 20 is a diagram illustrating a fourth example of the virtual space screen according to the embodiment.[FIG. 21]FIG. 21 is a diagram illustrating a fifth example of the virtual space screen according to the embodiment.[FIG. 22]FIG. 22 is a diagram illustrating the fifth example of the virtual space screen according to the embodiment.[FIG. 23]FIG. 23 is a diagram illustrating a sixth example of the virtual space screen according to the embodiment.[FIG. 24]FIG. 24 is a diagram illustrating a seventh example of the virtual space screen according to the embodiment.[FIG. 25]FIG. 25 is a diagram illustrating input information according to the embodiment.[FIG. 26]FIG. 26 is a diagram illustrating an eighth example of the virtual space screen according to the embodiment.[FIG. 27]FIG. 27 is a flowchart illustrating an example of a process for generating a virtual space screen including designated additional information.The accompanying drawings are intended to depict embodiments of the present disclosure and should not be interpreted to limit the scope thereof. The accompanying drawings are not to be considered as drawn to scale unless explicitly noted. Also, identical or similar reference numerals designate identical or similar components throughout the several views.[Description of Embodiments]

[0008] In describing embodiments illustrated in the drawings, specific terminology is employed for the sake of clarity. However, the disclosure of this specification is not intended to be limited to the specific terminology so selected and it is to be understood that each specific element includes all technical equivalents that have a similar function, operate in a similar manner, and achieve a similar result. Referring now to the drawings, embodiments of the present disclosure are described below. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. In industries such as civil engineering and construction, BIM and CIM are becoming increasingly adopted to address the declining birthrate and aging population issues and improve labor productivity, for example.

[0009] BIM is an acronym for Building Information Modeling. BIM is a solution for making use of information in all processes involving design, construction, and maintenance and management of a building by using a database of structures in which attribute data such as cost, finish, and management information is added to a digital model of a three-dimensional building (hereinafter referred to as "3D model") created on a computer.

[0010] CIM is an acronym for Construction Information Modeling. CIM is proposed as an analogy of BIM, which has been increasingly used in the construction field, and is a solution for the civil engineering field (general infrastructure such as roads and power, gas, and water supplies). Like BIM, CIM is an approach to share information, including a 3D model, among parties involved to improve the efficiency and sophistication of a series of construction production systems.

[0011] One issue to promote BIM and CIM is how to easily construct a 3D model of a building or a public facility.

[0012] BIM and CIM are relatively easy to achieve in the case of construction of a new structure, because the structure to be completed can be modeled from scratch using 3D computer aided design (CAD) software. In the case of an existing building, in contrast, BIM and CIM are difficult to achieve because, for example, the original blueprint, which was used to construct the building, may be missing or the building may differ from the original blueprint due torenovations. The BIM of such an existing building is referred to, for example, as-built BIM. As-built BIM is a significant challenge for BIM and CIM promotions in the future.

[0013] One method for implementing as-built BIM is a workflow of measuring a space using a laser scanner (hereinafter referred to as "LS") and creating a CAD model from a measured point cloud. In the related art, a space is measured using a photograph or a measure, and sketched to restore the space, which incurs a large amount of operation cost. The use of the LS greatly increases operating efficiency.

[0014] The as-built BIM using the LS facilitates modeling, but involves an operation called point cloud processing, which is not included in the workflow of the related art. Typical point cloud processing involves measuring multiple points using the LS, aligning point clouds to form an integrated point cloud, from which an unnecessary point cloud such as noise is removed, and finally converting the point cloud into a CAD model.

[0015] The operations described above are performed using an automatic model creation function of commercially available point cloud processing software. In the commercially available point cloud processing software, however, when objects of different categories, such as a pipe, a desk, and a person, are present in the same space, that is, the same point cloud, the three- dimensional shapes of the objects of the different categories may be collectively generated with insufficient accuracy, or the processing may take a long time.

[0016] When a three-dimensional shape of an object in a single category is generated, the generation process is repeatedly performed a number of times equal to the number of categories to be used.

[0017] In view of the problems described above, an embodiment discloses a technique for appropriately generating a three-dimensional shape with accuracy desired by a user without undue processing time or effort.

[0018] Another problem is how to utilize a constructed BIM and / or CIM model. Specifically, three- dimensional virtual spaces restored by BIM and CIM can be utilized for design and construction purposes and other operations such as maintenance and management and field inspection. That is, BIM and CIM may be used for purposes other than blueprints, such as making a record in a three-dimensional virtual space or sharing the record with another person. In this case, the three-dimensional virtual space has a larger amount of information than the two-dimensional virtual space, and is difficult to handle at a glance.

[0019] The technique according to the embodiment makes it possible to appropriately display additional information for an article in a three-dimensional virtual space screen, and makes the three-dimensional virtual space screen easier to handle.

[0020] FIG. l is a general arrangement diagram of a three-dimensional shape generation system according to an embodiment of the present disclosure. A three-dimensional shape processing system 1 according to the present embodiment includes a terminal apparatus 3, which is an example of an input / output apparatus, and a server 5.

[0021] The server 5 is an example of an information processing apparatus connected to the input / output apparatus via a network. The server 5 is also an example of a three-dimensional shape generation apparatus that generates three-dimensional shape information indicating a three-dimensional shape corresponding to a three-dimensional point cloud by using model shape information indicating a three-dimensional model shape.

[0022] The three-dimensional point cloud is a set of coordinate points in a virtual three-dimensional space that can be handled by, for example, a computer. The three-dimensional point cloud may also be referred to as a point cloud. The three-dimensional point cloud is a set of coordinate points corresponding to measurement points on the surface of an object when a space in which the object is present is measured using, for example, the LS. Color information may be added to each coordinate point, and red (R), green (G), and blue (B) values of each coordinate point may be added as the color information.

[0023] In the example described above, the three-dimensional point cloud is measured using the LS. Any other optical measurement means or mechanical measurement means may be used to measure the three-dimensional point cloud. Examples of the optical measurement means include a method using a stereo camera and a method using visual simultaneous localization and mapping (visual SLAM).

[0024] The three-dimensional shape information is information indicating a three-dimensional shape of an object that can be handled by, for example, a computer. The information indicating the three-dimensional shape is information that can geometrically identify the three-dimensional shape. For example, in the case of a sphere, the coordinates of the center of the sphere and the radius of the sphere correspond to the information indicating the three-dimensional shape. When the three-dimensional shape of an object is represented by a polyhedron (polygon), the coordinate points of the vertices of the polyhedron are an example of the information indicating the three-dimensional shape. In another example, any information that can uniquely define the shape of an object may be used as the information indicating the three-dimensional shape.

[0025] The three-dimensional shape information may include information indicating the three- dimensional shape of an object and information on the color or material of the object.

[0026] The three-dimensional model shape is a model such as a prototype or a template used to generate three-dimensional shape information from a three-dimensional point cloud. The model shape information is information indicating a three-dimensional model shape, and one item of model shape information corresponds to one three-dimensional model shape.

[0027] In some examples, the model shape information includes information related to the color or material of a three-dimensional model, in addition to the three-dimensional model shape. Specifically, a three-dimensional model shape indicating a plane may be provided with information called texture, such as the color or pattern of a wall, or information such as the material of the wall.

[0028] The three-dimensional model shape provided with texture information facilitates the representation of a CAD model with a color or a pattern. The three-dimensional model shape provided with material information allows a CAD model that can be provided with material information to have the material information as information as is, without the user setting the information.

[0029] As used herein, the term "texture" originally refers to the texture or touch of the surface of an object. In the three-dimensional shape information, a texture refers to a pattern or an image to be attached to the surface of a three-dimensional shape to represent the surface quality of the object.

[0030] CAD models typically include a "surface" model, a "solid" model, and a "polygon" model. The "solid" model has information on the appearance and information on the content such as the material, which allows calculation of the mass and the volume and creation of a crosssection shape. Thus, the "solid" model most realistically represents an actual object.

[0031] Setting, for example, a color, a material, brightness, and a background in a CAD model is referred to as "rendering", and rendering enables creation of a beautiful image like a photograph.

[0032] As described above, the three-dimensional shape information includes a three-dimensional geometric shape handled in 3D CAD, BIM, and CIM or polygon information such as geometric information, a mesh object, and texture.

[0033] The terminal apparatus 3 and the server 5 can communicate with each other via a communication network 100. The communication network 100 includes the Internet, a mobile communication network, and a local area network (LAN), for example. The communication network 100 may include a wired communication network and a wireless communication network. The wireless communication network may be based on a wireless communication standard such as third generation (3G), Worldwide Interoperability for Microwave Access (WiMAX), or Long Term Evolution (LTE). In one embodiment, the terminal apparatus 3 performs communication using short-range communication technology such as near field communication (NFC®).

[0034] Hardware ConfigurationFIG. 2 is a hardware configuration diagram of the terminal apparatus 3 and the server 5 according to the present embodiment. The hardware components of the terminal apparatus 3 are denoted by reference numerals in 300 series. The hardware components of the server 5 are denoted by reference numerals in 500 series in parentheses.

[0035] The terminal apparatus 3 includes a central processing unit (CPU) 301, a read only memory (ROM) 302, a random access memory (RAM) 303, a hard disk (HD) 304, a hard disk drive (HDD) 305, a recording medium 306, a medium interface (I / F) 307, a display 308, a network I / F 309, a keyboard 311, a mouse 312, a compact disc-rewritable (CD-RW) drive 314, and a bus line 310.

[0036] The CPU 301 controls the entire operation of the terminal apparatus 3. The ROM 302 stores a program used for driving the CPU 301. The RAM 303 is used as a work area for the CPU 301. The HD 304 stores various types of data such as programs. The HDD 305 controls reading or writing of various types of data from or to the HD 304 under control of the CPU 301. The medium I / F 307 controls reading or writing (storing) of data from or to the recording medium 306 such as a flash memory. The display 308 displays various types of information such as a cursor, a menu, a window, text, or an image. The network I / F 309 is an interface for performing data communication using the communication network 100. The keyboard 311 is an example of an input means that includes a plurality of keys for inputting characters, numerical values, various instructions, and so on. The mouse 312 is an example of an input means that allows the user to select or execute various instructions, select a target for processing, or move a cursor being displayed. The CD-RW drive 314 controls reading or writing of various types of data from or to a CD-RW 313, which is an example of a removable recording medium.

[0037] The server 5 includes a CPU 501, a ROM 502, a RAM 503, an HD 504, an HDD 505, a recording medium 506, a medium I / F 507, a display 508, a network I / F 509, a keyboard 511, amouse 512, a CD-RW drive 514, and a bus line 510. Since the components of the server 5 described above are the same or substantially the same as the components of the terminal apparatus 3 described above (i.e., the CPU 301, the ROM 302, the RAM 303, the HD 304, the HDD 305, the recording medium 306, the medium I / F 307, the display 308, the network I / F 309, the keyboard 311, the mouse 312, the CD-RW drive 314, and the bus line 310), a description thereof will be omitted.

[0038] In place of the CD-RW drive 314 (514), any other drive such as a compact disc-recordable (CD-R) drive may be used. In one example, each of the terminal apparatus 3 and the server 5 is implemented by a single computer. In another example, each of the terminal apparatus 3 and the server 5 is implemented by multiple computers to which any one or more of the components (functions, means, or storage units) are allocated as desired.

[0039] FIG. 3 is a functional block diagram of the three-dimensional shape generation system according to the present embodiment.

[0040] As illustrated in FIG. 3, the terminal apparatus 3 includes a transmitting and receiving unit 31, a reception unit 32, a display control unit 34, a determination unit 36, and a storing and reading unit 39. Each of these units is a function or means implemented by or caused to function by any one of the hardware components illustrated in FIG. 2 operating in accordance with instructions from the CPU 301 according to a program loaded onto the RAM 303 from the HD 304. The terminal apparatus 3 further includes a storage unit 3000, which is implemented by the RAM 303 and the HD 304 illustrated in FIG. 2.

[0041] Functional Components of Terminal ApparatusNext, the components of the terminal apparatus 3 will be described.

[0042] The transmitting and receiving unit 31 is an example of transmitting means and is implemented by instructions from the CPU 301 and by the network I / F 309 illustrated in FIG. 2. The transmitting and receiving unit 31 transmits and receives various types of data (or information) to and from other terminals, apparatuses, or systems via the communication network 100.

[0043] The reception unit 32 is an example of reception means and is mainly implemented by instructions from the CPU 301 and by the keyboard 311 and the mouse 312 illustrated in FIG. 2. The reception unit 32 receives various inputs from the user.

[0044] The display control unit 34 is an example of display control means and is implemented by instructions from the CPU 301 illustrated in FIG. 2. The display control unit 34 controls the display 308, which is an example of a display unit, to display various images or a screen.

[0045] The determination unit 36 is an example of determination means and is implemented by instructions from the CPU 301 illustrated in FIG. 2. The determination unit 36 performs various determinations.

[0046] The storing and reading unit 39 is an example of storage control means and is implemented by instructions from the CPU 301 and by the HDD 305, the medium I / F 307, and the CD-RW drive 314 illustrated in FIG. 2. The storing and reading unit 39 performs a process of storing various types of data in the storage unit 3000, the recording medium 306, and the CD-RW 313 and reading various types of data from the storage unit 3000, the recording medium 306, and the CD-RW 313.

[0047] Functional Configuration of ServerThe server 5 includes a transmitting and receiving unit 51, a processing unit 53, a determination unit 55, a generation unit 56, a setting unit 57, and a storing and reading unit 59. Each of these units is a function or means implemented by or caused to function by any one of the hardware components illustrated in FIG. 2 operating in accordance with instructions from the CPU 501 according to a program loaded onto the RAM 503 from the HD 504. The server 5 further includes a storage unit 5000, which is implemented by the HD 504 illustrated in FIG. 2. The storage unit 5000 is an example of storage means.

[0048] Functional Components of ServerNext, the components of the server 5 will be described. The server 5 may be configured such that one or more functions thereof are implemented by multiple computers in a distributed manner. In the following, the server 5 is described as a server computer residing on a cloud environment. In some embodiments, the server 5 may be a server residing in an on-premises environment.

[0049] The transmitting and receiving unit 51 is an example of transmitting means and is implemented by instructions from the CPU 501 and by the network I / F 509 illustrated in FIG. 2. The transmitting and receiving unit 51 transmits and receives various types of data (or information) to and from other terminals, apparatuses, or systems via the communication network 100.

[0050] The processing unit 53 is an example of processing means and is implemented by instructions from the CPU 501 illustrated in FIG. 2. The processing unit 53 performs various processesdescribed below. The processing unit 53 is an example of a three-dimensional information generation means that generates three-dimensional shape information.

[0051] The determination unit 55 is an example of determination means and is implemented by instructions from the CPU 501 illustrated in FIG. 2. The determination unit 55 performs various determinations described below.

[0052] The generation unit 56 is an example of generation means and is implemented by instructions from the CPU 501 illustrated in FIG. 2. The generation unit 56 generates various screens.

[0053] The setting unit 57 is an example of setting means and is implemented by instructions from the CPU 501 illustrated in FIG. 2. The setting unit 57 performs various settings and determinations described below.

[0054] The storing and reading unit 59 is an example of storage control means and is implemented by instructions from the CPU 501 and by the HDD 505, the medium I / F 507, and the CD-RW drive 514 illustrated in FIG. 2. The storing and reading unit 59 performs a process of storing various types of data in the storage unit 5000, the recording medium 506, and a CD-RW 513 and reading various types of data from the storage unit 5000, the recording medium 506, and the CD-RW 513. The storage unit 5000, the recording medium 506, and the CD-RW 513 are examples of a storage means.

[0055] The storage unit 5000 stores a setting information management database (DB) 5001, which is implemented by a setting information management table, a storage processing management DB 5002, a point cloud management DB 5003, a three-dimensional shape management DB 5004, a model shape management DB 5005, and a virtual space management DB 5006, which is implemented by a virtual space management table.

[0056] The setting information management DB 5001 stores and manages various types of information. The storage processing management DB 5002 stores and manages various processing programs for generating a three-dimensional shape. The point cloud management DB 5003 stores and manages three-dimensional point cloud information for generating a three-dimensional shape. The three-dimensional shape management DB 5004 stores and manages three-dimensional shape information generated using model shape information. The model shape management DB 5005 stores and manages model shape information indicating a three-dimensional model shape for each item of model identification information. The virtual space management DB 5006 stores and manages information on a virtual space including the three-dimensional shape.

[0057] FIG. 4 is an illustration of an example of a setting information management table according to the present embodiment.

[0058] The setting information management table is a table for managing three-dimensional point cloud data for generating a three-dimensional shape and a history of processes for generating the three-dimensional shape. In the storage unit 5000, the setting information management DB 5001 includes the setting information management table as illustrated in FIG. 4. The setting information management table manages, for each user ID, which is an example of information on a user, a file name of three-dimensional point cloud data, a history of processes for generating a three-dimensional shape, and attribute information indicating the attribute of the user in association with each other.

[0059] FIG. 5 is an illustration of an example of the virtual space management table according to the present embodiment.

[0060] The virtual space management table is a table for managing information on a virtual space including three-dimensional shape information. In the storage unit 5000, the virtual space management DB 5006 includes the virtual space management table as illustrated in FIG. 5.

[0061] The virtual space management table manages, for each item of virtual space identification information identifying a virtual space, model identification information, position information, article information, attribute information, dimension-line position information, and dimension information in association with each other.

[0062] The model identification information is information identifying model shape information indicating a three-dimensional model shape. The position information is information indicating the position of the three-dimensional model shape in the virtual space in the form of three-dimensional coordinates X, Y, and Z.

[0063] The article information is information on an article indicated by the three-dimensional shape including a single or plurality of model shapes. The article indicated by the three-dimensional shape is an example of a target portion in a spatial image indicating a space.

[0064] The attribute information is information identifying the attribute of a user who is an operator of a three-dimensional shape displayed on the display 308.

[0065] The dimension-line position information is information defining the position information of a dimension line for a specific three-dimensional shape using three-dimensional coordinates X, Y, and Z. The dimension line includes a line in one direction indicating one dimension, suchas (Xa, Ya, Za) to (Xb, Yb, Zb), lines in two directions indicating two dimensions, such as (XI, Yl, Zl) to (X2, Y2, Z2) and (X3, Y3, Z3) to (X4, Y4, Z4), and lines in three directions indicating three dimensions.

[0066] The dimension information is information indicating a dimension corresponding to a dimension line in terms of a numerical value. In a case where the dimension line is a line in one direction, the length is indicated by a numerical value. In a case where the dimension line is a line in two directions, at least one of the length and the area is indicated by a numerical value. In a case where the dimension line is a line in three directions, at least one of the volume or capacity, the length, and the area is indicated by a numerical value. When the dimension information indicates the area, the target surface may be solidly colored instead of using a dimension line. The dimension-line position information and the dimension information are examples of additional information for a target portion in a spatial image.

[0067] FIG. 6 is a sequence diagram illustrating an example of a three-dimensional shape generation process according to the present embodiment.

[0068] The reception unit 32 of the terminal apparatus 3 receives an operation of inputting a user ID of a user on an input / output screen displayed on the display 308 (step SI). The transmitting and receiving unit 31 transmits a request for a setting screen including the user ID received in step SI to the server 5, and the transmitting and receiving unit 51 of the server 5 receives the request transmitted from the terminal apparatus 3 (step S2).

[0069] Then, the storing and reading unit 59 of the server 5 searches the setting information management DB 5001 using the user ID included in the request received in step S2 as a search key to read a file name of three-dimensional point cloud data associated with the user ID included in the request and a history of processes for generating a three-dimensional shape, and the generation unit 56 of the server 5 generates a setting screen based on the file name and the history read by the storing and reading unit 59 (step S3).

[0070] The setting screen includes a model information setting screen, a point cloud setting screen, and a processing setting screen. The model information setting screen receives a model setting operation for setting a plurality of items of model shape information to be used to generate three-dimensional shape information, from among a plurality of items of model shape information. The point cloud setting screen receives a point cloud setting operation for setting a three-dimensional point cloud to be used to generate three-dimensional shape information. The processing setting screen receives a processing setting operation for setting a processing program for generating three-dimensional shape information.

[0071] The processing setting screen includes the display of a process initially set based on the history read by the storing and reading unit 59.

[0072] The transmitting and receiving unit 51 transmits setting screen information for the setting screen generated in step S3 to the terminal apparatus 3, and the transmitting and receiving unit 31 of the terminal apparatus 3 receives the setting screen information transmitted from the server 5 (step S4). Step S4 is an example of a transmission step of transmitting model information setting screen information for the model information setting screen. The transmitting and receiving unit 31 is an example of transmitting means.

[0073] Then, the display control unit 34 of the terminal apparatus 3 controls the display 308 to display the setting screen received in step S4 (step S5). The reception unit 32 of the terminal apparatus 3 receives a predetermined input operation performed by the user on the displayed setting screen. The input operation includes a model setting operation for setting a plurality of items of model shape information to be used to generate three-dimensional shape information, from among a plurality of items of model shape information, a point cloud setting operation, and a processing setting operation. Step S5 is an example of a reception step of receiving a model setting operation for setting a plurality of items of model shape information to be used to generate three-dimensional shape information, from among a plurality of items of model shape information.

[0074] The transmitting and receiving unit 31 transmits input information corresponding to the input operation received by the reception unit 32 to the server 5, and the transmitting and receiving unit 51 of the server 5 receives the input information transmitted from the terminal apparatus 3 (step S6). The input information includes model setting information for setting a plurality of items of model shape information to be used to generate three-dimensional shape information, from among a plurality of items of model shape information, point cloud setting information, and processing setting information.

[0075] The storing and reading unit 59 of the server 5 updates the history of processes for generating the three-dimensional shape stored in the setting information management DB 5001 in association with the user ID, based on the processing setting information and the model setting information included in the input information received in step S6 (step S7).

[0076] Then, the storing and reading unit 59 of the server 5 searches the point cloud management DB 5003 using the point cloud setting information included in the input information received in step S6 as a search key to read the three-dimensional point cloud data associated with the point cloud setting information.

[0077] The storing and reading unit 59 further searches the storage processing management DB 5002 using the processing setting information included in the input information received in step S6 as a search key to read the processing program associated with the processing setting information.

[0078] Then, the storing and reading unit 59 searches the model shape management DB 5005 using the model identification information corresponding to the model setting information included in the input information received in step S6 as a search key to read the three-dimensional model shape associated with the model identification information.

[0079] The processing unit 53 of the server 5 generates three-dimensional shape information, based on the three-dimensional point cloud data, the processing program, and the three-dimensional model shape read from the storing and reading unit 59 (step S8).

[0080] Step S8 is an example of a three-dimensional information generation step of generating three- dimensional shape information, based on a model setting operation on the model information setting screen and point cloud information indicating a three-dimensional point cloud, by using a plurality of items of model shape information set by the model setting operation.

[0081] The generation unit 56 of the server 5 generates an operation screen for receiving an operation input for generating three-dimensional shape information, and the transmitting and receiving unit 51 transmits operation screen information for the operation screen to the terminal apparatus 3 (step S9).

[0082] The transmitting and receiving unit 31 of the terminal apparatus 3 receives the operation screen information transmitted from the server 5, the display control unit 34 of the terminal apparatus 3 controls the display 308 to display the operation screen indicated by the received operation screen information, and the reception unit 32 of the terminal apparatus 3 receives a predetermined input operation performed by the user on the displayed operation screen (step S10).

[0083] The input operation includes a shape setting operation for setting one of a plurality of items of three-dimensional shape information.

[0084] The transmitting and receiving unit 31 transmits input information corresponding to the input operation received by the reception unit 32 to the server 5, and the transmitting and receiving unit 51 of the server 5 receives the input information transmitted from the terminal apparatus 3 (step Sil).

[0085] The input information includes shape setting information obtained by the shape setting operation, and the processing unit 53 of the server 5 determines three-dimensional shape information, based on the shape setting information included in the input information received in step S 11.

[0086] The terminal apparatus 3 and the server 5 repeatedly perform steps S8 to Sil, as appropriate.

[0087] The processing unit 53 converts the generated three-dimensional shape information into, for example, a CAD format, and the storing and reading unit 59 stores the converted three- dimensional shape information in the three-dimensional shape management DB 5004, the recording medium 506, or the CD-RW 513 in association with model identification information of a single or plurality of model shapes used to generate the three-dimensional shape and position information in the three-dimensional point cloud data read in step S8 (step S12). The CAD format is three-dimensional shape information that can be handled with 3D CAD. In a case where the three-dimensional shape information is handled with a commercially available 3D CAD tool, the three-dimensional shape information is converted into a 3D CAD format.

[0088] The storing and reading unit 59 updates the history of processes for generating the three- dimensional shape stored in the setting information management DB 5001 in association with the user ID, based on the model shape information corresponding to the determined three- dimensional shape information (step SI 3).

[0089] Then, the storing and reading unit 59 of the server 5 reads all of the items of three- dimensional shape information generated based on the three-dimensional point cloud data read in step S8 from the three-dimensional shape management DB 5004 together with the model identification information and the position information, and the generation unit 56 of the server 5 generates a virtual space screen including the items of three-dimensional shape information read by the storing and reading unit 59 (step S14). The virtual space screen is a screen indicating a space corresponding to the three-dimensional point cloud data read in step S8 as a virtual space including a single or plurality of three-dimensional shapes generated based on the three-dimensional point cloud data.

[0090] The transmitting and receiving unit 51 transmits virtual space screen information indicating the virtual space screen generated in step S14 to the terminal apparatus 3 (step S15).

[0091] The transmitting and receiving unit 31 of the terminal apparatus 3 receives the virtual space screen information transmitted from the server 5, and the display control unit 34 of theterminal apparatus 3 controls the display 308 to display the virtual space screen indicated by the received virtual space screen information (step SI 6).

[0092] The reception unit 32 of the terminal apparatus 3 receives a predetermined input operation performed by the user on the displayed virtual space screen. The input operation includes an operation of setting the virtual space identification information, the model identification information, the position information, the article information, the attribute information, the dimension-line position information, and the dimension information illustrated in FIG. 5.

[0093] The transmitting and receiving unit 31 transmits input information corresponding to the input operation received by the reception unit 32 to the server 5, and the transmitting and receiving unit 51 of the server 5 receives the input information transmitted from the terminal apparatus 3 (step SI 7). The input information includes the virtual space identification information, the model identification information, the position information, the article information, the attribute information, the dimension-line position information, and the dimension information.

[0094] The storing and reading unit 59 of the server 5 stores the virtual space identification information, the model identification information, the position information, the article information, the attribute information, the dimension-line position information, and the dimension information included in the input information received in step S17 in the virtual space management DB 5006 in association with the model identification information and the position information read in step S14 (step SI 8). As a result, the various types of information illustrated in FIG. 5 are stored in the virtual space management DB 5006.

[0095] In the foregoing, the functions of the server 5 illustrated in FIG. 3 may be integrated into the terminal apparatus 3, and the terminal apparatus 3 may perform the processes of the server 5 illustrated in FIG. 6.

[0096] FIG. 7 is a sequence diagram illustrating an example of a virtual space operation process according to the present embodiment.

[0097] The reception unit 32 of the terminal apparatus 3 receives an operation of inputting virtual space identification information on the input / output screen displayed on the display 308 (step S101).

[0098] The transmitting and receiving unit 31 transmits the virtual space identification information input by the operation received by the reception unit 32 in step S 101 to the server 5, and the transmitting and receiving unit 51 of the server 5 receives the virtual space identification information transmitted from the terminal apparatus 3 (step SI 02).

[0099] The storing and reading unit 59 of the server 5 searches the virtual space management DB 5006 using the virtual space identification information received in step SI 02 as a search key to read the model identification information, the position information, and the article information associated with the virtual space identification information, and searches the model shape management DB 5005 using the model identification information read from the virtual space management DB 5006 as a search key to read the three-dimensional model shape associated with the model identification information.

[0100] The generation unit 56 of the server 5 generates a virtual space screen, based on the model identification information, the three-dimensional model shape, the position information, and the article information read by the storing and reading unit 59 (step SI 03).

[0101] The transmitting and receiving unit 51 transmits virtual space screen information indicating the virtual space screen generated in step SI 03 to the terminal apparatus 3 (step SI 04).

[0102] The transmitting and receiving unit 31 of the terminal apparatus 3 receives the virtual space screen information transmitted from the server 5, and the display control unit 34 of the terminal apparatus 3 controls the display 308 to display the virtual space screen indicated by the received virtual space screen information (step SI 05).

[0103] The reception unit 32 of the terminal apparatus 3 receives a predetermined input operation performed by the user on the displayed virtual space screen and an operation on the three- dimensional shape included in the virtual space screen. The input operation includes an operation of inputting a user ID. The input operation may further include use information.

[0104] The transmitting and receiving unit 31 transmits input information corresponding to the input operation received by the reception unit 32 to the server 5, and the transmitting and receiving unit 51 of the server 5 receives the input information transmitted from the terminal apparatus 3 (step SI 06). The input information includes the user ID, designation information designating a type of dimension information, operation information for changing the position and orientation of a virtual camera located at the position of a point of view from which the virtual space is viewed, and specification information specifying an article.

[0105] The storing and reading unit 59 of the server 5 searches the setting information management DB 5001 using the user ID received in step SI 06 as a search key to read the attribute information associated with the user ID, and searches the virtual space management DB 5006 using the article information, the designation information, and the specification information received in step SI 06 and the attribute information read from the setting informationmanagement DB 5001 as search keys to read the dimension-line position information and the dimension information associated with the article information, the designation information, the specification information, and the attribute information.

[0106] The generation unit 56 of the server 5 superimposes the dimension-line position information and the dimension information read by the storing and reading unit 59 on the virtual space screen that is generated in step SI 03 and for which the point of view is changed based on the operation information received in step SI 06 to generate a virtual space screen (step SI 07).

[0107] Additional information such as the dimension-line position information and the dimension information is not limited to information stored in association with the attribute information. In another example, information calculated based on the three-dimensional shape information may be used as additional information. In one embodiment, the processing unit 53 calculates dimension-line position information, based on the three-dimensional shape information. In another embodiment, the processing unit 53 calculates dimension information, based on read dimension-line position information or calculated dimension-line position information.

[0108] The transmitting and receiving unit 51 transmits virtual space screen information indicating the virtual space screen generated in step SI 07 to the terminal apparatus 3 (step SI 08).

[0109] The transmitting and receiving unit 31 of the terminal apparatus 3 receives the virtual space screen information transmitted from the server 5, and the display control unit 34 of the terminal apparatus 3 controls the display 308 to display the virtual space screen indicated by the received virtual space screen information (step SI 09). The processing of steps SI 06 to SI 09 described above is repeatedly performed.

[0110] In the foregoing, the functions of the server 5 illustrated in FIG. 3 may be integrated into the terminal apparatus 3, and the terminal apparatus 3 may perform the processes of the server 5 illustrated in FIG. 7.

[0111] FIG. 8 is an illustration of a display screen according to the present embodiment.

[0112] FIG. 8 illustrates a display screen 1000 displayed on the display 308 of the terminal apparatus 3 in step S5 of the sequence diagram illustrated in FIG. 6.

[0113] The display control unit 34 of the terminal apparatus 3 causes a user information display screen 1100, a setting screen 1200, and a confirmation button 1300, which is an example of an instruction reception screen, to be displayed on the display screen 1000.

[0114] The setting screen 1200 includes a point cloud setting screen 1210, a processing setting screen 1220, and a model information setting screen (an example of a reception screen) 1230.

[0115] The point cloud setting screen 1210 is a screen for receiving a point cloud setting operation for setting point cloud information indicating a three-dimensional point cloud to be used to generate three-dimensional shape information. The display control unit 34 causes point cloud setting boxes 1212 and 1214 to be displayed in association with the respective file names of a plurality of items of point cloud data read by the storing and reading unit 59. In one embodiment, more than one of the point cloud setting boxes 1212 and 1214 is set.

[0116] The processing setting screen 1220 is a screen for receiving a processing setting operation for setting a processing program for generating three-dimensional shape information. The display control unit 34 causes processing setting boxes 1222, 1224, and 1226 to be displayed in association with the respective names of a plurality of processes.

[0117] In FIG. 8, the processing setting box 1222 is used to set a processing program for registration, the processing setting box 1224 is used to set a processing program for noise removal, and the processing setting box 1226 is used to set a processing program for segmentation. The details of the processing programs will be described below. The processing setting box 1222 for registration may be automatically set when a plurality of point clouds are set in the point cloud setting screen 1210. In one example, the processing setting box 1226 for segmentation may be omitted since segmentation processing is performed without exception.

[0118] The display control unit 34 initially sets the settings of the processing setting boxes 1222, 1224, and 1226, based on the history read by the storing and reading unit 59 and causes the processing setting boxes 1222, 1224, and 1226 to be displayed. If no history is present, predetermined settings are displayed as initial settings.

[0119] The model information setting screen 1230 is an example of a reception screen for receiving a model setting operation for setting a plurality of items of model shape information to be used to generate three-dimensional shape information, from among a plurality of items of model shape information.

[0120] The model information setting screen 1230 includes model information setting boxes 1231, 1234, and 1235, candidate model setting boxes 1232 and 1233, and a model display screen 1240.

[0121] The display control unit 34 causes the model information setting boxes 1231, 1234, and 1235 to be displayed in association with the respective names of a plurality of items of modelinformation, and causes the candidate model setting boxes 1232 and 1233 to be displayed in association with the respective names of a plurality of model shapes. As used herein, the term "model information" refers to a model shape information group including one or more items of model shape information.

[0122] In FIG. 8, the model information setting box 1231 is used to set model information indicating a three-dimensional model shape of a pipe, the model information setting box 1234 is used to set model information indicating a three-dimensional model shape of a desk, and the model information setting box 1235 is used to set model information indicating a three-dimensional model shape of a person.

[0123] The candidate model setting box 1232 is used to set model shape information indicating the three-dimensional model shape of model A of the pipe, and the candidate model setting box 1233 is used to set model shape information indicating the three-dimensional model shape of model B of the pipe.

[0124] The display control unit 34 causes a candidate model 1242, a candidate model 1243, or a model shape 1244 to be displayed on the model display screen 1240 in association with the name of the set model information or candidate model. In a case where the model shape information includes a color, the candidate model 1242 or 1243 or the model shape 1244 may be displayed with color. In one embodiment, each of a plurality of candidate models includes different three-dimensional model shapes to be used to generate a plurality of items of three- dimensional shape information from the same area in the three-dimensional point cloud. Of the generated plurality of items of three-dimensional shape information, one item of three- dimensional shape information is determined in the subsequent process.

[0125] The candidate model 1242 indicates a plurality of three-dimensional model shapes of the model A of the pipe. The candidate model 1243 indicates a plurality of three-dimensional model shapes of the model B of the pipe. The model shape 1244 indicates a three-dimensional model shape of the desk.

[0126] In response to various setting boxes being pointed at with a pointing device such as the mouse 312, the display control unit 34 of the terminal apparatus 3 displays a check mark in each of the various setting boxes, and the reception unit 32 receives various setting operations. In response to the confirmation button 1300 being operated, the reception unit 32 of the terminal apparatus 3 confirms the various setting operations.

[0127] Then, as described in step S6 in FIG. 6, the transmitting and receiving unit 31 transmits, to the server 5, input information including various types of setting information corresponding to the various setting operations received by the reception unit 32.

[0128] FIG. 9 is a flowchart illustrating an example of a three-dimensional shape generation process according to the present embodiment, and illustrates a process corresponding to step S8 in FIG. 6.

[0129] The processing unit 53 of the server 5 acquires the three-dimensional point cloud data read from the storing and reading unit 59 (step S21). In a case where a plurality of items of three- dimensional point cloud data are acquired and the processing program read from the storing and reading unit 59 includes a processing program for registration, the processing unit 53 performs a registration process (step S22). The registration process is a process for converting a plurality of three-dimensional point clouds into one integrated three-dimensional point cloud.

[0130] In a case where the processing program read from the storing and reading unit 59 includes a processing program for noise removal, the processing unit 53 performs a noise removal process (step S23). The noise removal process is a process for removing an unnecessary point cloud from the three-dimensional point cloud.

[0131] The setting unit 57 of the server 5 sets model information, based on the model setting information included in the input information received in step S6 in FIG. 6 (step S24). For example, as illustrated in FIG. 8, when the model setting information includes model information indicating the three-dimensional model shape of the pipe and model information indicating the three-dimensional model shape of the desk, the setting unit 57 first sets the model information indicating the three-dimensional model shape of the pipe.

[0132] The setting unit 57 sets one or more items of model shape information, based on the model setting information included in the input information received in step S6 in FIG. 6 (step S25). For example, as illustrated in FIG. 8, when the model setting information includes, as a plurality of candidate models, model shape information indicating the three-dimensional model shape of the model A of the pipe and model shape information indicating the three- dimensional model shape of the model B of the pipe, the setting unit 57 first sets the model shape information indicating the three-dimensional model shape of the model A of the pipe.

[0133] In a case where the processing program read from the storing and reading unit 59 includes a processing program for segmentation, the processing unit 53 performs a segmentation process (step S26).

[0134] The segmentation process is also referred to as a point cloud sorting process, and one known technique is machine learning Point Transformer. Point Transformer is described in X-F. Han, Y.-J. Kuang, and G.-Q. Xiao, "Point Cloud Learning with Transformer," 2021, arXiv:2104.13636.

[0135] The segmentation process is a process for labeling a specific point cloud in a three- dimensional point cloud to distinguish the specific point cloud from the other point clouds in the three-dimensional point cloud. In one example, a plurality of specific point clouds are differently labeled to distinguish the plurality of specific point clouds from each other. In another example, the segmentation process may be performed together with a clustering process for grouping point clouds close to each other in the labeled point cloud.

[0136] The processing unit 53 checks the point cloud labeled in step S26 against each of the plurality of three-dimensional model shapes included in the model shape information set in step S25, and replaces a specific area corresponding to the point cloud with a three-dimensional model shape most similar to the shape of the point cloud among the three-dimensional model shapes (step S27). In one example, when none of the model shapes is optimum, the processing unit 53 adjusts the dimensions or shapes of the model shapes. In a case where texture information may be different for the same shape, models having the same shape but different textures may be also used to check the point cloud. In one example, the processing unit 53 may skip the segmentation process in step S26 and perform the processing of step S27 after step S25.

[0137] The determination unit 55 of the server 5 determines whether any unprocessed point cloud that is yet to be subjected to replacement is present (step S28). If an unprocessed point cloud is present (YES at S28), the process returns to step S27. In step S27, the processing unit 53 performs a replacement process on the unprocessed point cloud to replace a specific area using the three-dimensional model shape. When the ratio of unprocessed point clouds to the entire point cloud is less than or equal to a predetermined value, the determination unit 55 may determine that no unprocessed point cloud is present, that is, the processing of the point clouds is completed.

[0138] On the other hand, if no unprocessed point cloud is present (NO at S28), the determination unit 55 determines whether any unprocessed candidate model is present (step S29). If an unprocessed candidate model is present (YES at S29), the process returns to step S25, and the setting unit 57 sets the model shape information included in the unprocessed candidate model. For example, the setting unit 57 sets, as a plurality of candidate models, the model shape information indicating the three-dimensional model shape of the model B of the pipe aftersetting the model shape information indicating the three-dimensional model shape of the model A of the pipe.

[0139] If no unprocessed candidate model is present (NO at S29), the processing unit 53 determines three-dimensional shape information, based on the shape setting information included in the input information received in step Sil illustrated in FIG. 6 (step S30). For example, the setting unit 57 determines, as three-dimensional shape information, one of the plurality of candidate models, namely, the three-dimensional shape information generated based on the model shape information of the model A of the pipe and the three-dimensional shape information generated based on the model shape information of the model B of the pipe.

[0140] The determination unit 55 determines whether any unprocessed model information is present (step S31). If no unprocessed model information is present (NO at S31), the process ends.

[0141] If an unprocessed model information is present (YES at S31), the process returns to step S24, and the setting unit 57 sets the unprocessed model information. For example, the setting unit 57 sets model information of the desk after setting the model information of the pipe.

[0142] FIGs. 10A and 10B are illustrations of a registration process according to the present embodiment.

[0143] FIG. 10A illustrates a first three-dimensional point cloud 1410, and FIG. 10B illustrates a second three-dimensional point cloud 1420.

[0144] In one example, the processing unit 53 aligns the position of a feature point 1410a included in the first three-dimensional point cloud 1410 with the position of a feature point 1420a included in the second three-dimensional point cloud 1420 to convert the first three- dimensional point cloud 1410 and the second three-dimensional point cloud 1420 into one integrated three-dimensional point cloud.

[0145] FIG. 11 is an illustration of a noise removal process according to the present embodiment. The processing unit 53 removes an unnecessary point cloud 1440 from a three-dimensional point cloud 1430.

[0146] FIG. 12 is an illustration of a segmentation process according to the present embodiment.

[0147] The processing unit 53 labels a specific point cloud in the three-dimensional point cloud 1430 to form a labeled point cloud 1450 (black circles) to make the labeled point cloud 1450 distinguishable from the other point clouds.

[0148] FIGs. 13A, 13B, and 13C are illustrations of a model collation and replacement process according to the present embodiment.

[0149] FIG. 13 A illustrates the three-dimensional point cloud 1430. FIG. 13B illustrates candidate models 1460 of pipe A. FIG. 13C illustrates candidate models 1470 of pipe B.

[0150] The candidate models 1460 of the pipe A include a plurality of model shapes 1461, 1462, and 1463, each having a different shape. The candidate models 1470 of the pipe B include a plurality of model shapes 1471, 1472, and 1473, each having a different shape.

[0151] The processing unit 53 checks the labeled point cloud 1450 (point cloud labeled with black circles) against each of the plurality of model shapes 1461, 1462, and 1463 of the pipe A, and replaces a specific area corresponding to the labeled point cloud 1450 with a model shape most similar to the shape of the labeled point cloud 1450 among the model shapes 1461, 1462, and 1463. In one example, when none of the model shapes 1461, 1462, and 1463 is optimum, the processing unit 53 adjusts the dimensions or shapes of the model shapes 1461, 1462, and 1463. In one example, the processing unit 53 determines a model shape most similar to the shape of a point cloud by calculating distances between respective points in the point cloud and corresponding portions of each of model shapes and determining a model shape for which the sum of the distances is smallest among the model shapes.

[0152] In a case where textures or materials may be different for the same shape, a plurality of textures or a plurality of materials may be also used to check the target point cloud. For example, the colors (RGB) of the point cloud are compared with the colors of the texture included in the model shape information, and the closest color is set. Specifically, the closest color is determined by, for example, but not limited to, determining differences between the color information (RGB) of the texture for the model having the portion closest to each point in the point cloud and the RGB values of the point cloud and calculating the sum of the absolute values of the differences.

[0153] Further, the processing unit 53 checks the labeled point cloud 1450 (point cloud labeled with black circles) against each of the plurality of model shapes 1471, 1472, and 1473 of the pipe B, and replaces the specific area corresponding to the labeled point cloud 1450 with a model shape most similar to the shape of the labeled point cloud 1450 among the model shapes 1471, 1472, and 1473. In one example, when none of the model shapes is optimum, the processing unit 53 adjusts the dimensions or shapes of the model shapes. In one example, when none of the model shapes 1471, 1472, and 1473 is optimum, the processing unit 53 adjusts the dimensions or shapes of the model shapes 1471, 1472, and 1473.

[0154] FIG. 14 is an illustration of an operation screen according to the present embodiment.

[0155] FIG. 14 illustrates a display screen 1000 displayed on the display 308 of the terminal apparatus 3 in step S10 of the sequence diagram illustrated in FIG. 6.

[0156] The display control unit 34 of the terminal apparatus 3 causes an operation screen 1500 and a confirmation button 1300 to be displayed on the display screen 1000. The operation screen 1500 includes a generated shape setting screen 1510 and a generated shape display screen 1560.

[0157] The generated shape setting screen 1510 is an example of a second reception screen for receiving a shape setting operation for setting one of a plurality of items of three-dimensional shape information, and includes generated shape setting boxes 1512 and 1514.

[0158] The generated shape setting box 1512 is used to set a three-dimensional shape generated based on the model shape information of the model A of the pipe. The generated shape setting box 1514 is used to set a three-dimensional shape generated based on the model shape information of the model B of the pipe.

[0159] The display control unit 34 causes generated shapes 1562 and 1564 to be displayed on the generated shape display screen 1560 in association with the respective names of the plurality of items of model shape information.

[0160] The generated shape 1562 indicates a three-dimensional shape generated based on the model shape information of the model A of the pipe. The generated shape 1564 indicates a three- dimensional shape generated based on the model shape information of the model B of the pipe.

[0161] In response to the generated shape setting box 1512 or 1514 being pointed at with a pointing device such as the mouse 312, the reception unit 32 of the terminal apparatus 3 receives a shape setting operation. The reception unit 32 confirms the shape setting operation in response to the confirmation button 1300 being operated.

[0162] Then, as described in step Sil in FIG. 6, the transmitting and receiving unit 31 transmits, to the server 5, input information including shape setting information obtained by the shape setting operation received by the reception unit 32. In FIG. 14, one of the generated shapes 1562 and 1564 is selected based on the operation by the user. In another example, the processing unit 53 of the server 5 may automatically select a generated shape more similar tothe shape of the labeled point cloud among the generated shapes 1562 and 1564 without being based on an operation by the user. In one example, a generated shape more similar to the shape of a point cloud is determined by calculating distances between respective points in the point cloud and corresponding portions of each of generated shapes and determining a generated shape for which the sum of the distances is smaller.

[0163] FIG. 15A is a flowchart illustrating an example of a virtual space operation process according to the present embodiment, and illustrates a process performed by the server 5 and corresponding to step SI 07 in the sequence diagram illustrated in FIG. 7.

[0164] The generation unit 56 of the server 5 generates a virtual space screen by changing the point of view for the virtual space screen generated in step SI 03 based on the operation information received in step SI 06 (step S41).

[0165] The determination unit 55 determines whether an article included in the virtual space screen is an article for which additional information is to be displayed, that is, an article to be hidden, based on the input information received in step SI 06 (step S42). Specifically, when an article is specified based on the input information, the determination unit 55 determines the article to be a display target. When the article is not specified based on the input information, the determination unit 55 does not determine the article to be a display target. If none of the articles is specified based on the input information, the determination unit 55 determines all of the articles to be display targets.

[0166] The input information includes at least one of a user ID identifying a user and specification information specifying an article. In response to receipt of the user ID, the article corresponding to the user ID is specified, as described in step SI 07.

[0167] Accordingly, for an article that is likely to be of interest to the user and is specified based on a user ID or specification information, additional information is displayed. Therefore, additional information can be appropriately displayed in accordance with the interest of the user.

[0168] The generation unit 56 generates a virtual space screen on which, for an article determined not to be a display target in step S42, that is, an article determined to be hidden (YES at S42), no dimension lines or dimensions are displayed in a superimposed manner (step S43).

[0169] The determination unit 55 determines whether the article to be displayed extends beyond the virtual space screen (step S44). More specifically, as illustrated in FIG. 15B, the determination unit 55 determines that the article to be displayed extends beyond the virtualspace screen when the amount or percentage by which the article to be displayed extends beyond the virtual space screen is greater than or equal to a threshold.

[0170] FIG. 15B is a flowchart illustrating an example of a process for determining whether an article extends off-screen according to the present embodiment. First, the determination unit 55 determines whether the angle of view at which the virtual space screen is actually displayed on the display (for example, the display 308) is unknown. If the angle of view is unknown (step S440: Yes), the process proceeds to step S441. On the other hand, if the angle of view is not unknown (step S440: No), the process proceeds to step S442.

[0171] If the angle of view is unknown, the determination unit 55 sets virtual point-of-view coordinates (the coordinates of the position of a point of view from which the virtual space is viewed) and estimates an angle of view, based on the virtual point-of-view coordinates (step S441). Specifically, the determination unit 55 calculates the angle of view at which the virtual space screen is displayed on the display in the vertical direction and the horizontal direction from the set virtual point-of-view coordinates. In one example, a plurality of sets of virtual point-of-view coordinates may be set to estimate an angle of view, and an appropriate angle of view may be selected according to the estimation result.

[0172] Then, the determination unit 55 calculates virtual point-of-view coordinates, based on a given angle of view if the angle of view is not unknown, or based on the angle of view estimated in step S441 if the angle of view is unknown (step S442). Then, the determination unit 55 calculates a three-dimensional space corresponding to the virtual space screen, based on the angle of view and the virtual point-of-view coordinates (step S443). Specifically, the determination unit 55 uses virtual point-of-view coordinates O and four vertices A, B, C, and D of the virtual space screen to calculate a three-dimensional space formed by four half lines OA, OB, OC, and OD.

[0173] If the amount by which the article extends beyond the three-dimensional space (the amount of extension) is less than or equal to a threshold (step S444: Yes), the determination unit 55 determines that the article does not extend beyond the virtual space screen (step S445). On the other hand, if the amount of extension is not less than or equal to the threshold (step S444: No), the determination unit 55 determines that the article extends beyond the virtual space screen (step S446). The amount of extension is represented by, for example, the number of sets of coordinates out of the three-dimensional space among sets of coordinates representing the shape of the article. When the article is a cube, the sets of coordinates representing the shape of the article are the coordinates of the eight vertices of the article. When the article is a triangular prism, the sets of coordinates representing the shape of the article are the coordinates of the six vertices of the article. When the shape of the article is represented bytriangular meshes, the sets of coordinates representing the shape of the article is a collection of sets of coordinates of the vertices of the triangular meshes. The sets of coordinates representing the shape of the article may be any collection of sets of coordinates other than the sets of coordinates of the vertices described above as long as the collection of sets of coordinates can represent the shape of the article.

[0174] The threshold used in step S444 is, for example, a value set in advance by experiment. For example, when the threshold is 0, the determination unit 55 determines that the article extends beyond the screen if even one of the sets of coordinates representing the shape of the article is out of the three-dimensional space. When the threshold is 2, the determination unit 55 determines that the article extends beyond the screen if three or more of the sets of coordinates representing the shape of the article are out of the three-dimensional space.

[0175] The generation unit 56 generates a virtual space screen on which, for an article determined to extend beyond the screen in step S44 (Yes at S44), no dimension lines or dimensions are displayed in a superimposed manner (step S43).

[0176] The determination unit 55 determines whether the visually recognizable area of the surface of the article to be displayed exceeds a threshold (step S45).

[0177] The generation unit 56 generates a virtual space screen on which, for an article for which the visually recognizable area is determined to be less than or equal to the threshold in step S45 (NO at S45), no dimension lines or dimensions are displayed in a superimposed manner (step S43).

[0178] The processing unit 53 calculates the distance from the position of the virtual camera to the article to be displayed (step S46). The determination unit 55 determines whether the distance from the virtual camera to the article to be displayed is less than a threshold (step S47).

[0179] When the virtual space screen is a two-dimensional image, the processing unit 53 performs image analysis on the two-dimensional image to calculate the distance from an image capturing position to the article to be displayed.

[0180] The generation unit 56 generates a virtual space screen on which, for an article for which the distance is determined to be greater than or equal to the threshold in step S47 (NO at S47), no dimension lines or dimensions are displayed in a superimposed manner (step S43).

[0181] The determination unit 55 determines whether the article to be displayed is arranged at the center of the virtual space screen (step S48). If the article to be displayed is arranged at thecenter (YES at S48), the generation unit 56 generates a virtual space screen on which all the dimension lines and dimensions stored in the virtual space management DB 5006 in association with the article are displayed in a superimposed manner (step S49).

[0182] On the other hand, if the article to be displayed is not arranged at the center (NO at S48), the generation unit 56 generates a virtual space screen on which some of the dimension lines and dimensions stored in the virtual space management DB 5006 in association with the article are displayed in a superimposed manner (step S50).

[0183] In step S50, the generation unit 56 may generate a virtual space screen on which no dimension lines or dimensions are displayed in a superimposed manner.

[0184] In one embodiment, if it is determined in step S44 that at least one of the sets of coordinates representing the shape of the article is out of the three-dimensional space, the generation unit 56 hides all of the items of additional information related to the article. In this case, if a portion of the article (such as two points forming a side of the shape of the article or four points forming a surface of the shape of the article) is in the three-dimensional space, the generation unit 56 may display additional information related to the portion.

[0185] In another embodiment, the determination unit 55 may perform two threshold determinations in each of steps S44, S45, and S47, and the generation unit 56 may generate a virtual space in any one of steps S43, S49, and S50. For example, at step S47, the generation unit 56 may generate a virtual space as described in step S43, when the determination unit 55 determines that the distance from the virtual camera to the article to be displayed is equal to or less than a first threshold, or is equal or greater than a second threshold. The first threshold is a lower threshold, which is less than the second threshold that may be referred to as an upper threshold. In other words, the article that is too close to the virtual camera is not displayed, as well as the article that is far from the virtual camera.

[0186] After the processing of steps S43, S49, and S50 in FIG. 15 A, in response to at least one of the position, the orientation, and the angle of view of the virtual camera being changed by an input operation of the user, the process returns to step S41. At this time, the generation unit 56 may generate a virtual space screen in accordance with information obtained after the change is made, the determination unit 55 may perform the determination of steps S42, S44, S45, S47, and S48 in accordance with the information obtained after the change is made, and the generation unit 56 may perform the processing of step S43, S49, or S50.

[0187] That is, in a case where an article for which additional information is displayed extends beyond the virtual space screen due to the change of at least one of the position, theorientation, and the angle of view of the virtual camera, the generation unit 56 generates a virtual space screen that displays the additional information such that the display of, for example, a dimension and a dimension line is changed in accordance with information obtained after the change is made. In one example, the generation unit 56 generates a virtual space screen that displays additional information (post-change additional information) different from additional information displayed before the change is made (pre-change additional information) in at least one of the number and type of items of information, or a virtual space screen that displays neither the pre-change additional information nor the postchange additional information. In another example, in a case where an article for which additional information is not displayed due to an extension beyond the virtual space screen appears within the virtual space screen after the change is made, the generation unit 56 generates a virtual space screen that displays the additional information related to the article.

[0188] FIGs. 16A, 16B, and 16C are illustrations of a three-dimensional image according to the present embodiment.

[0189] FIG. 16A is a view illustrating positions of a virtual camera IC and a predetermined area T in a case in which a spherical image is of a three-dimensional solid sphere. The virtual camera IC corresponds to a position of a point of view (viewpoint) of a user who is viewing a spherical image CE represented as a surface area of a three-dimensional solid sphere.

[0190] FIG. 16B is a perspective view of the virtual camera IC and the predetermined area T illustrated in FIG. 16A. FIG. 16C illustrates a predetermined-area image displayed on the display.

[0191] In FIG. 16B, the spherical image CE illustrated in FIG. 16A is represented by a three- dimensional sphere CS. Assuming that the spherical image CE generated in the way described above is a surface area of the sphere CS, as illustrated in FIG. 16B, the virtual camera IC is located inside the spherical image CE.

[0192] The predetermined area T in the spherical image CE is an imaging area of the virtual camera IC. Specifically, the predetermined area T is specified by predetermined-area information indicating an image capturing direction and an angle of view of the virtual camera IC in a three-dimensional virtual space containing the spherical image CE.

[0193] In one embodiment, zooming in or out of the predetermined area T is implemented by bringing the virtual camera IC closer to or farther away from the spherical image CE. A predetermined-area image Q is an image of the predetermined area T in the spherical imageCE. The predetermined area T is defined by the angle of view of the virtual camera IC and the distance from the virtual camera IC to the spherical image CE.

[0194] That is, the display control unit 34 changes the position and the orientation of the virtual camera IC located at the position of the point of view from which the three-dimensional image is viewed to change the display area of the three-dimensional image to be displayed on the display 308.

[0195] In the example described above, the three-dimensional image to be displayed on the display 308 is a spherical image. In another example, three-dimensional point cloud data may be used. A three-dimensional point cloud is arranged in a virtual space, and a virtual camera is arranged in the virtual space. The three-dimensional point cloud is projected onto a predetermined projection surface in the virtual space, based on predetermined-area information indicating the point-of-view position, the image capturing direction, and the angle of view of the virtual camera, to obtain a three-dimensional image. The point-of-view position and the orientation of the virtual camera are changed to change the display area of the three- dimensional image.

[0196] FIG. 17 is a diagram illustrating a virtual space screen according to the present embodiment.

[0197] FIG. 17 illustrates a display screen 1000 displayed on the display 308 of the terminal apparatus 3 in step SI 09 of the sequence diagram illustrated in FIG. 7.

[0198] The display control unit 34 of the terminal apparatus 3 causes a virtual space screen 1600 to be displayed on the display screen 1000. The virtual space screen 1600 includes a beam 1611, columns 1613 and 1615, and walls 1620 and 1630, which are examples of an article including a three-dimensional shape, and is viewed from the virtual camera IC described in FIGs. 16A to 16C.

[0199] The display control unit 34 causes dimension lines 1711A and 1711B, dimensions 1811A and 181 IB, a target surface 171 IC, and a dimension (area) 181 IC to be displayed on the beam 1611 in a superimposed manner.

[0200] The display control unit 34 causes dimension lines 1713A and 1713B and dimensions 1813A and 1813B to be displayed on the column 1613 in a superimposed manner. The display control unit 34 causes dimension lines 1715 A and 1715B and dimensions 1815A and 1815B to be displayed on the column 1615 in a superimposed manner.

[0201] The display control unit 34 does not cause dimension lines or dimensions to be displayed on the walls 1620 and 1630 in a superimposed manner.

[0202] That is, the generation unit 56 generates the display screen 1000 such that all of the dimension lines and the dimensions for the beam 1611 are displayed and no dimension lines or dimensions are displayed for the wall 1630 to which the distance from the position of the point of view is different from and longer than that to the beam 1611.

[0203] Accordingly, the user can grasp different additional information for each of the target portions with different distances from the point of view.

[0204] In one example, the generation unit 56 may generate the display screen 1000 such that some of the dimension lines and dimensions of the wall 1630 are displayed.

[0205] FIG. 18 is a diagram illustrating a second example of the virtual space screen according to the present embodiment.

[0206] As in FIG. 17, the display control unit 34 of the terminal apparatus 3 causes a virtual space screen 1600 including the beam 1611, the columns 1613 and 1615, and the walls 1620 and 1630 to be displayed on the display screen 1000.

[0207] Unlike the display screen 1000 illustrated in FIG. 17, on the display screen 1000 illustrated in FIG. 18, a target surface 1613C of the column 1613 is not visually recognizable, and the dimension line 1713B and the dimension 1813B corresponding to the width of the target surface 1613C are not displayed.

[0208] That is, the column 1613 includes the target surface 1613C, and the generation unit 56 generates the display screen 1000 such that additional information is displayed for the target surface 1613C, based on the size of the target surface 1613C in the virtual space screen 1600, when the size of the target surface 1613C is greater than or equal to a threshold.

[0209] Accordingly, the additional information is displayed when the target surface 1613C is visually recognizable, and the additional information is not displayed when the target surface 1613C is difficult to visually recognize. Therefore, the user can appropriately grasp a target surface for which the additional information is being displayed.

[0210] Unlike the display screen 1000 illustrated in FIG. 17, on the display screen 1000 illustrated in FIG. 18, furthermore, the beam 1611 extends beyond the virtual space screen 1600, and noneof the dimension lines 1711A and 1711B, the dimensions 1811A and 181 IB, the target surface 1711C, and the dimension (area) 1811C is displayed for the beam 1611.

[0211] That is, the generation unit 56 generates the display screen 1000 such that additional information is displayed for an article, based on an extension of the article beyond the virtual space screen 1600, when the amount or percentage of the extension is less than a threshold.

[0212] Accordingly, additional information is not displayed for an article that is less likely to be of interest to the user and extends beyond the virtual space screen 1600. Therefore, additional information can be appropriately displayed in accordance with the interest of the user.

[0213] FIG. 19 is a diagram illustrating a third example of the virtual space screen according to the present embodiment.

[0214] The display control unit 34 of the terminal apparatus 3 causes a virtual space screen 1600 including the walls 1620 and 1630 to be displayed on the display screen 1000.

[0215] Unlike the display screen 1000 illustrated in FIG. 17, on the display screen 1000 illustrated in FIG. 19, the display control unit 34 causes dimension lines 1720A, 1720B, 1720C, and 1720D, dimensions 1820 A, 1820B, 1820C, and 1820D, and a dimension (area) 1820E to be displayed on the wall 1620 in a superimposed manner.

[0216] Likewise, the display control unit 34 causes dimension lines 1730A and 1730B and dimensions 1830A and 1830B to be displayed on the wall 1630 in a superimposed manner. The dimension lines 1730A and 1730B and the dimensions 1830A and 1830B correspond to an opening 1635.

[0217] That is, the generation unit 56 generates the display screen 1000 such that, as illustrated in FIG. 19, all of the dimension lines and the dimensions are displayed for the wall 1630 when the distance from the position of the point of view to the wall 1630 is a first distance shorter than a threshold and, as illustrated in FIG. 17, no dimension lines or dimensions are displayed for the wall 1630 when the distance from the position of the point of view to the wall 1630 is a second distance different from the first distance and longer than the threshold.

[0218] Accordingly, different additional information can be displayed in a case where the distance from the point of view to a target portion changes.

[0219] In one example, the generation unit 56 may generate the display screen 1000 such that some of the dimension lines and the dimensions for the wall 1630 are displayed when the distance from the position of the point of view to the wall 1630 is the second distance.

[0220] FIG. 20 is a diagram illustrating a fourth example of the virtual space screen according to the present embodiment.

[0221] The display control unit 34 of the terminal apparatus 3 causes a virtual space screen 1600 including the wall 1630 to be displayed on the display screen 1000.

[0222] Unlike the display screen 1000 illustrated in FIG. 19, on the display screen 1000 illustrated in FIG. 20, the wall 1630 does not extend beyond the virtual space screen 1600. The display control unit 34 further causes dimension lines 1730C, 1730D, 1730E, and 1730F and dimensions 1830C, 1830D, 1830E, and 1830F to be displayed on the wall 1630 in a superimposed manner.

[0223] FIG. 21 is a diagram illustrating a fifth example of the virtual space screen according to the present embodiment.

[0224] The display control unit 34 of the terminal apparatus 3 causes a virtual space screen 1600 including the walls 1620 and 1630 to be displayed on the display screen 1000.

[0225] That is, the generation unit 56 generates the display screen 1000 such that the dimension lines 1730A and 1730B and the dimensions 1830A and 1830B corresponding to the opening 1635 are displayed in a superimposed manner on the wall 1630, which is arranged at the center, but no dimension lines or dimensions are displayed in a superimposed manner on the wall 1620, which is not arranged at the center.

[0226] FIG. 22 is a diagram illustrating the fifth example of the virtual space screen according to the present embodiment.

[0227] The display control unit 34 of the terminal apparatus 3 causes a virtual space screen 1600 including the walls 1620 and 1630 to be displayed on the display screen 1000.

[0228] That is, the generation unit 56 generates the display screen 1000 such that the dimension lines 1720B, 1720C and 1720D, the dimensions 1820B, 1820C, and 1820D, and the dimension (area) 1820E are displayed in a superimposed manner on the wall 1620, which is arranged at the center, but no dimension lines or dimensions are displayed in a superimposed manner on the wall 1630, which is not arranged at the center.

[0229] FIG. 23 is a diagram illustrating a sixth example of the virtual space screen according to the present embodiment.

[0230] As in FIG. 17, the display control unit 34 of the terminal apparatus 3 causes a virtual space screen 1600 including the beam 1611, the columns 1613 and 1615, and the walls 1620 and 1630 to be displayed on the display screen 1000. In addition, the display control unit 34 further causes pop-up screens 1910A, 1910B, 1910C, and 1910D to be displayed on the display screen 1000.

[0231] The pop-up screens 1910A, 1910B, 1910C, and 1910D are another example of additional information for an article and present images of the beam 1611 as viewed from top, left, right, and bottom, respectively.

[0232] FIG. 24 is a diagram illustrating a seventh example of the virtual space screen according to the present embodiment.

[0233] The display control unit 34 of the terminal apparatus 3 causes a virtual space screen 1600 including a wall 1640 to be displayed on the display screen 1000.

[0234] The display control unit 34 first causes a dimension line 1740 and a dimension 1840 to be displayed on the wall 1640 in a superimposed manner.

[0235] In this state, the user performs an operation of selecting the dimension line 1740 or the dimension 1840 using the reception unit 32. Then, the display control unit 34 further causes dimension lines 1740a and 1740b, dimensions 1840a and 1840b, a dimension (area) 1840c, and a comment balloon 1940 to be displayed on the wall 1640 in a superimposed manner. The comment balloon 1940 is another example of additional information for an article.

[0236] In one example, the display control unit 34 causes a comment on the dimension 1840 to be displayed in the comment balloon 1940. The comment on the dimension 1840 is, for example, a comment on how the dimension 1840 of the wall 1640 is set or a comment on the benefit of the set dimension 1840. In one embodiment, a comment (an example of additional information) on each target portion is created in advance and stored in, for example, the model shape management DB 5005. In this case, the storing and reading unit 59 reads a comment on a target portion selected by the user, and the display control unit 34 causes the read comment to be displayed on the target portion in a superimposed manner.In another embodiment, the processing unit 53 may use multimodal artificial intelligence (Al) to generate a comment on a target portion selected by the user. For example, multimodal Al may learn combinations of data of target portions and texts related to the target portions by contrastive learning, which is known as, for example, contrastive language image pre-training (CLIP). The data of each target portion is three-dimensional point cloud information or three- dimensional shape information corresponding to the target portion. The text related to each target portion is a text created by, for example, a designer or a skilled person and indicating the purpose of design or know-how of the target portion. In this case, the processing unit 53 uses the multimodal Al that has learned to generate a comment on a target portion selected by the user, and the display control unit 34 causes the generated comment to be displayed on the target portion in a superimposed manner.

[0238] FIG. 25 is a diagram illustrating input information according to the present embodiment.

[0239] The display control unit 34 of the terminal apparatus 3 causes a dimension type selection screen 1920 and an article selection screen 1930 to be displayed on the display screen 1000.

[0240] The generation unit 56 generates the display screen 1000 such that a dimension line and a dimension of a type specified in the dimension type selection screen 1920, which is received by the reception unit 32, are displayed for an article specified in the article selection screen 1930, which is received by the reception unit 32. Therefore, additional information can be appropriately displayed in accordance with the interest of the user.

[0241] The generation unit 56 may generate a virtual space screen that displays designated additional information described below instead of or in addition to the additional information described above. The designated additional information is displayed based on designation information received by the reception unit 32, whereas the additional information is displayed by changing at least one of the position, the orientation, and the angle of view of the virtual camera located at the virtual point-of-view coordinates. The designation information is input by, for example, the user designating a position in the virtual space screen or additional information being displayed. The designated additional information is, for example, designated dimension-line position information (dimension-line position information that is designated) or designated dimension information (dimension information corresponding to the designated dimensionline position information or dimension information that is designated). The designated dimension-line position information and the designated dimension information are examples of designated additional information.

[0242] In one example, the designation operation is an operation of designating the position of a three-dimensional shape included in the virtual space screen. In one embodiment, theprocessing unit 53 determines designated dimension-line position information, based on the designated position.

[0243] In another example, the designation operation is an operation of designating dimension-line position information or dimension information on a virtual space screen on which the dimension-line position information and the dimension information are superimposed. In one embodiment, the processing unit 53 determines, as the designated dimension-line position information or the designated dimension information, the dimension-line position information or dimension information that is designated.

[0244] The additional information and the designated additional information may be displayed in different formats. In one example, the additional information and the designated additional information are displayed in different colors. In another example, the additional information and the designated additional information are displayed in different sizes. Displaying the additional information and the designated additional information in different formats makes it easier to visually recognize the designated additional information.

[0245] When a portion of the target portion extends beyond the virtual space screen, the generation unit 56 may generate a virtual space screen that displays the designated additional information for the target portion without displaying the additional information for the target portion.

[0246] FIG. 26 is a diagram illustrating an eighth example of the virtual space screen according to the present embodiment. FIG. 26 illustrates an example in which the designated additional information is displayed on the virtual space screen 1600 illustrated in FIG. 18. The difference from FIG. 18 is that the position information of the dimension line 1711 A is designated as designated dimension-line position information and the dimension 1811 A is designated as designated dimension information. Since a portion of the beam 1611 extends beyond the virtual space screen 1600, the dimension line 1711 A and the dimension 1811 A are not displayed for the beam 1611 in FIG. 18. In FIG. 26, in contrast, a designated dimension line 1711A' and a designated dimension 1811A' are displayed for the beam 1611. In FIG. 26, the designated dimension line 1711 A' and the designated dimension 1811 A' are displayed in a format similar to that of the dimension line 1713 A and the dimension 1813 A. In another embodiment, the designated dimension line 1711 A' and the designated dimension 1811 A' may be displayed in different formats.

[0247] FIG. 27 is a flowchart illustrating an example of a process for generating a virtual space screen including designated additional information. First, the determination unit 55 determines whether additional information corresponding to an article is designated by the user (step S500). If the additional information is designated by the user (step S500: Yes), theprocess proceeds to step S501. On the other hand, if the additional information is not designated by the user (step S500: No), the process proceeds to step S502.

[0248] In step S501, even when a portion of the article extends beyond the virtual space screen, the generation unit 56 generates a virtual space screen that displays the additional information (designated additional information) corresponding to the article.

[0249] In step S502, if a portion of the article extends beyond the virtual space screen, the generation unit 56 generates a virtual space screen that does not display the additional information corresponding to the article.

[0250] As described above, even when a portion of a target portion extends beyond the virtual space screen, designated additional information is displayed. Accordingly, by designating additional information of high interest, the user can visually recognize the additional information of high interest even when a portion of an article extends beyond the virtual space screen due to a change of the point of view or the angle of view of the virtual space screen.

[0251] First AspectIn a first aspect, a server 5, which is an example of an information processing apparatus, or a terminal apparatus 3 integrated with the server 5 includes a generation unit 56. The generation unit 56 generates a display screen 1000 that displays a virtual space screen 1600 indicating a space and additional information for an article such that the additional information is superimposed on the virtual space screen 1600. The additional information is generated based on a distance from a position of a point of view from which the virtual space screen 1600 is viewed to the article in the virtual space screen 1600. The virtual space screen 1600 is an example of a spatial image. The generation unit 56 is an example of display screen generation means. The article is an example of a target portion.

[0252] Accordingly, a user can appropriately grasp the additional information for the article in accordance with the distance from the position of the point of view to the article.

[0253] That is, an article with a short distance from the position of the point of view has a large image area in the virtual space screen 1600 and is easy to visually recognize even in a case where additional information for the article is displayed, whereas an article with a long distance from the position of the point of view has a small image area in the virtual space screen 1600 and is difficult to visually recognize in a case where additional information for the article is displayed.

[0254] Accordingly, a large amount of additional information is displayed for an article that is likely to be of interest to the user and has a short distance from the position of the point of view, and a small amount of additional information or no additional information is displayed for an article that is less likely to be of interest to the user and has a long distance from the position of the point of view. Therefore, additional information can sufficiently and smoothly be displayed in accordance with the interest of the user.

[0255] Second AspectAccording to a second aspect, in the server 5 or the terminal apparatus 3 integrated with the server 5 of the first aspect, the additional information includes a dimension line and a dimension for the article. Accordingly, the user can appropriately grasp the dimension line and the dimension displayed on the display screen 1000.

[0256] Third AspectAccording to a third aspect, in the server 5 or the terminal apparatus 3 integrated with the server 5 of the first aspect or the second aspect, the virtual space screen 1600 includes a three- dimensional image, and the generation unit 56 changes at least one of a position, an orientation, and an angle of view of a virtual camera located at the position of the point of view to change a display area in which the three-dimensional image is to be displayed on the display screen 1000.

[0257] In this case, the processing unit 53 calculates the distance from the position of the virtual camera to the article to be displayed. In a case where the virtual space screen 1600 is a two- dimensional image, the processing unit 53 performs image analysis on the two-dimensional image to calculate the distance from an image capturing position to the article to be displayed.

[0258] Accordingly, the user can easily grasp the additional information for the article in the three- dimensional image.

[0259] Fourth AspectAccording to a fourth aspect, in the server 5 or the terminal apparatus 3 integrated with the server 5 of the third aspect, the generation unit 56 generates the display screen 1000 such that all of dimension lines and dimensions, which are an example of first additional information, are displayed for a wall 1630, which is an example of the article, when the distance from the position of the point of view to the wall 1630 is a first distance shorter than a threshold, and generates the display screen 1000 such that some of the dimension lines and dimensions, which are an example of second additional information different from the first additional information in at least one of a number and type of items of information, are displayed for the wall 1630 or no dimension lines or dimensions are displayed for the wall 1630 when thedistance from the position of the point of view to the wall 1630 is a second distance different from the first distance and longer than the threshold.

[0260] Accordingly, different additional information can be displayed in a case where the distance from the point of view to a target portion changes.

[0261] Fifth AspectAccording to a fifth aspect, the server 5 or the terminal apparatus 3 integrated with the server 5 of the third aspect further includes input reception means, the additional information is displayed in response to at least one of the position, the orientation, and the angle of view of the virtual camera being changed, and the generation unit 56 generates the display screen 1000 such that the additional information and designated additional information are displayed on the virtual space screen 1600 in a superimposed manner. The designated additional information is displayed based on designation information received by the input reception means.

[0262] Accordingly, the designated additional information can be displayed in accordance with a designation operation.

[0263] Sixth AspectAccording to a sixth aspect, in the server 5 or the terminal apparatus 3 integrated with the server 5 of any one of the first aspect to the fifth aspect, the generation unit 56 generates the display screen 1000 such that all dimension lines and dimensions, which are an example of first additional information, are displayed for a beam 1611, which is an example of a first article, and generates the display screen 1000 such that some dimension lines and dimensions, which are an example of second additional information different from the first additional information in at least one of a number and type of items of information, are displayed for a wall 1630, which is an example of a second article to which a distance from the position of the point of view is different from and longer than a distance to the beam 1611 from the position of the point of view, or no dimension lines and dimensions are displayed for the wall 1630.

[0264] Accordingly, the user can grasp different additional information for each of the target portions with different distances from the point of view.

[0265] Seventh AspectAccording to a seventh aspect, in the server 5 or the terminal apparatus 3 integrated with the server 5 of any one of the first aspect to the sixth aspect, a column 1613, which is an example of the article, includes a target surface 1613C, and the generation unit 56 generates the displayscreen 1000 such that the additional information is displayed for the target surface 1613C, based on a size of the target surface 1613C in the virtual space screen 1600, when the size of the target surface 1613C is greater than or equal to a threshold, and generates the display screen 1000 such that the additional information is not displayed for the target surface 1613C when the size of the target surface 1613C is less than the threshold.

[0266] Accordingly, the additional information is displayed when the target surface 1613C is visually recognizable, and the additional information is not displayed when the target surface 1613C is difficult to visually recognize. Therefore, the user can appropriately grasp a target surface for which the additional information is being displayed.

[0267] Eighth AspectAccording to an eighth aspect, in the server 5 or the terminal apparatus 3 integrated with the server 5 of any one of the first aspect to the seventh aspect, the generation unit 56 generates the display screen 1000 such that the additional information is displayed for the article, based on an extension of the article beyond the virtual space screen 1600, when an amount or percentage of the extension is less than a threshold, and generates the display screen 1000 such that the additional information is not displayed for the article when the amount or percentage of the extension is greater than or equal to the threshold.

[0268] Accordingly, additional information is not displayed for an article that is less likely to be of interest to the user and extends beyond the virtual space screen 1600. Therefore, additional information can be appropriately displayed in accordance with the interest of the user.

[0269] Ninth AspectAccording to a ninth aspect, in the server 5 or the terminal apparatus 3 integrated with the server 5 of any one of the first aspect to the eighth aspect, the generation unit 56 generates the display screen 1000 such that in a case where a target portion for which third additional information is displayed extends beyond the virtual space screen 1600, fourth additional information different from the third additional information in at least one of a number and type of items of information is displayed or none of the third additional information and the fourth additional information is displayed.

[0270] Accordingly, it is possible to display additional information for which the number of dimensions or dimension lines to be displayed is changed in accordance with a change of the virtual space screen 1600.

[0271] Tenth AspectAccording to a tenth aspect, the server 5 or the terminal apparatus 3 integrated with the server 5 of any one of the first aspect to the ninth aspect further includes a reception unit 32, which is an example of input reception means, and the generation unit 56 generates the display screen 1000 such that additional information is displayed for an article specified based on input information received the reception unit 32.

[0272] Accordingly, for an article that is likely to be of interest to the user and is specified based on the input information, additional information is displayed. Therefore, additional information can be appropriately displayed in accordance with the interest of the user.

[0273] Eleventh AspectAccording to an eleventh aspect, in the server 5 or the terminal apparatus 3 integrated with the server 5 of the tenth aspect, the input information includes at least one of a user ID, which is an example of user information identifying a user, and specification information specifying the article. Therefore, additional information can be appropriately displayed in accordance with the interest of the user.

[0274] Twelfth AspectAccording to a twelfth aspect, the server 5 or the terminal apparatus 3 integrated with the server 5 of any one of the first aspect to the eleventh aspect further includes a reception unit 32, and the generation unit 56 generates the display screen 1000 such that additional information specified in at least type or number, based on input information received by the reception unit 32, is displayed for the article.

[0275] Therefore, additional information can be appropriately displayed in accordance with the interest of the user.

[0276] Thirteenth AspectIn a thirteenth aspect, a terminal apparatus 3, which is an example of an information processing apparatus, includes a display control unit 34. The display control unit 34 controls a display 308 to display a display screen 1000 that displays a virtual space screen 1600 indicating a space and additional information for an article such that the additional information is superimposed on the virtual space screen 1600. The additional information is generated based on a distance from a position of a point of view from which the virtual space screen 1600 is viewed to the article in the virtual space screen 1600.

[0277] Fourteenth AspectIn a fourteenth aspect, an information processing method is executed by a server 5, which is an example of an information processing apparatus, or a terminal apparatus 3 integrated withthe server 5. The information processing method includes generating (SI 07) a display screen 1000 that displays a virtual space screen 1600 indicating a space and additional information for an article such that the additional information is superimposed on the virtual space screen 1600. The additional information is generated based on a distance from a position of a point of view from which the virtual space screen 1600 is viewed to the article in the virtual space screen 1600.

[0278] Fifteenth AspectIn a fifteenth aspect, a program causes a computer to execute generating a display screen 1000 that displays a virtual space screen 1600 indicating a space and additional information for an article such that the additional information is superimposed on the virtual space screen 1600. The additional information is generated based on a distance from a position of a point of view from which the virtual space screen 1600 is viewed to the article in the virtual space screen 1600.

[0279] Sixteenth AspectIn a sixteenth aspect, a three-dimensional shape processing system 1, which is an example of an information processing system, includes a terminal apparatus 3 and a server 5 connected to the terminal apparatus 3 via a communication network 100. The server 5 includes a generation unit 56 and a transmitting and receiving unit 51. The generation unit 56 generates a display screen 1000 that displays a virtual space screen 1600 indicating a space and additional information for an article such that the additional information is superimposed on the virtual space screen 1600. The additional information is generated based on a distance from a position of a point of view from which the virtual space screen 1600 is viewed to the article in the virtual space screen 1600. The transmitting and receiving unit 51 transmits display screen information indicating the display screen 1000 to the terminal apparatus 3. The terminal apparatus 3 includes a transmitting and receiving unit 31 and a display control unit 34. The transmitting and receiving unit 31 receives the display screen information transmitted from the server 5. The display control unit 34 controls a display unit to display the display screen 1000.

[0280] The above-described embodiments are illustrative and do not limit the present invention. Thus, numerous additional modifications and variations are possible in light of the above teachings. For example, elements and / or features of different illustrative embodiments may be combined with each other and / or substituted for each other within the scope of the present invention. Any one of the above-described operations may be performed in various other ways, for example, in an order different from the one described above.

[0281] The present invention can be implemented in any convenient form, for example using dedicated hardware, or a mixture of dedicated hardware and software. The present invention may be implemented as computer software implemented by one or more networked processing apparatuses. The processing apparatuses include any suitably programmed apparatuses such as a general purpose computer, a personal digital assistant, a Wireless Application Protocol (WAP) or third-generation (3G)-compliant mobile telephone, and so on. Since the present invention can be implemented as software, each and every aspect of the present invention thus encompasses computer software implementable on a programmable device. The computer software can be provided to the programmable device using any conventional carrier medium (carrier means). The carrier medium includes a transient carrier medium such as an electrical, optical, microwave, acoustic or radio frequency signal carrying the computer code. An example of such a transient medium is a Transmission Control Protocol / Internet Protocol (TCP / IP) signal carrying computer code over an IP network, such as the Internet. The carrier medium may also include a storage medium for storing processor readable code such as a floppy disk, a hard disk, a compact disc read-only memory (CD- ROM), a magnetic tape device, or a solid state memory device.

[0282] The functionality of the elements disclosed herein may be implemented using circuitry or processing circuitry which includes general purpose processors, special purpose processors, integrated circuits, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and / or combinations thereof which are configured or programmed, using one or more programs stored in one or more memories, to perform the disclosed functionality. Processors are considered processing circuitry or circuitry as they include transistors and other circuitry therein. In the disclosure, the circuitry, units, or means are hardware that carry out or are programmed to perform the recited functionality. The hardware may be any hardware disclosed herein which is programmed or configured to carry out the recited functionality. There is a memory that stores a computer program which includes computer instructions. These computer instructions provide the logic and routines that enable the hardware (e.g., processing circuitry or circuitry) to perform the method disclosed herein. This computer program can be implemented in known formats as a computer-readable storage medium, a computer program product, a memory device, a record medium such as a CD- ROM or DVD, and / or the memory of an FPGA or ASIC.

[0283] This patent application is based on and claims priority to Japanese Patent Application No. 2023-169530, filed on September 29, 2023, and 2024-122596, filed on July 29, 2024, in the Japan Patent Office, the entire disclosure of which is hereby incorporated by reference herein. [Reference Signs List]

[0284] 1 three-dimensional shape processing system100 communication network3 terminal apparatus (example of input / output apparatus)31 transmitting and receiving unit (example of transmitting means and receiving means)32 reception unit (example of reception means)34 display control unit (example of display control means)36 determination unit39 storing and reading unit (example of storage control means)308 display (example of display unit)3000 storage unit5 server (example of three-dimensional shape generation apparatus)51 transmitting and receiving unit (example of transmitting means and receiving means)53 processing unit (example of three-dimensional information generation means)55 determination unit56 generation unit (example of display screen generation means)57 setting unit59 storing and reading unit (example of storage control means)5000 storage unit5001 setting information management DB (example of setting information management means)5002 storage processing management DB (example of storage processing management means)5003 point cloud management DB (example of point cloud management means)5004 three-dimensional shape management DB (example of three-dimensional shape management means)5005 model shape management DB5006 virtual space management DB1000 display screen1100 user information display screen1200 setting screen1210 point cloud setting screen1212, 1214 point cloud setting box1220 processing setting screen1222, 1224, 1226 processing setting box1230 model information setting screen (example of reception screen)1231, 1234, 1235 model information setting box1232, 1233 candidate model setting box1240 model display screen1242, 1243 candidate model1244 model shape1300 confirmation button (example of instruction reception screen)1410 first three-dimensional point cloud1420 second three-dimensional point cloud1430 three-dimensional point cloud1440 unnecessary point cloud1450 labeled point cloud1460, 1470 candidate model1461, 1462, 1463, 1471, 1472, 1473 model shape1500 operation screen1510 generated shape setting screen (example of second reception screen)1512, 1514 generated shape setting box1560 generated shape display screen1562, 1564 generated shape1600 virtual space screen (example of spatial image)1611 beam (example of target portion and article)1613, 1615 column (example of target portion and article)1620, 1630, 1640 wall (example of target portion and article)1711 to 1740 dimension line (example of additional information)1711 A' designated dimension line (example of designated additional information)1811 to 1840 dimension (example of additional information)1811 A' designated dimension (example of designated additional information)

Claims

[CLAIMS]1. An information processing apparatus comprising: display screen generation means configured to generate a display screen including a spatial image and additional information for a target portion, the spatial image being an image indicating a space including the target portion, the additional information being superimposed on the spatial image, wherein the additional information is generated based on a distance from a position of a point of view from which the spatial image is viewed to the target portion in the spatial image.

2. The information processing apparatus according to claim 1, wherein the additional information includes a dimension line and a dimension for the target portion.

3. The information processing apparatus according to claim 1 or 2, wherein the spatial image includes a three-dimensional image, and the display screen generation means is configured to change at least one of a position, an orientation, and an angle of view of a virtual camera located at the position of the point of view to change a display area in the display screen, the display area being an area in which the three-dimensional image is to be displayed.

4. The information processing apparatus according to claim 3, wherein the additional information includes first additional information in a case where the distance from the position of the point of view to the target portion is a first distance, and the additional information includes second additional information different from the first additional information in at least one of a number and type of items of information in a case where the distance from the position of the point of view to the target portion is a second distance different from the first distance.

5. The information processing apparatus according to claim 3, wherein the display screen generation means is configured to: generate the display screen such that the display screen includes the additional information in a case where the distance from the position of the point of view to the target portion is a first distance; and generate a display screen that does not include the additional information in a case where the distance from the position of the point of view to the target portion is a second distance different from the first distance.

6. The information processing apparatus according to claim 3, whereinthe display screen generation means is configured to generate the display screen such that, in addition to the additional information, designated additional information to be displayed based on input designation information is superimposed on the spatial image.

7. The information processing apparatus according to any one of claims 1 to 6, wherein the target portion includes a first target portion and a second target portion different from the first target portion in a distance from the position of the point of view, and the display screen generation means is configured to: generate, as the display screen, a first display screen including first additional information for the first target portion; and generate, as the display screen, a second display screen including second additional information for the second target portion, the second additional information being different from the first additional information in at least one of a number and type of items of information.

8. The information processing apparatus according to any one of claims 1 to 7, wherein the target portion includes a target surface, and the display screen generation means is configured to generate the display screen such that the additional information is displayed for the target surface, based on a size of the target surface in the spatial image.

9. The information processing apparatus according to any one of claims 1 to 8, wherein the display screen generation means is configured to generate the display screen such that the additional information is displayed for the target portion, based on an extension of the target portion beyond the spatial image.

10. The information processing apparatus according to any one of claims 1 to 9, wherein the display screen generation means is configured to generate the display screen such that another additional information different from the additional information in at least one of a number and type of items of information is displayed in a case where the target portion for which the additional information is displayed on the display screen extends beyond the spatial image.

11. The information processing apparatus according to any one of claims 1 to 10, whereinthe display screen generation means is configured to generate the display screen such that the additional information is displayed for the target portion, the target portion being specified based on input information that is input.

12. The information processing apparatus according to claim 11, wherein the input information includes at least one of user information identifying a user and specification information specifying the target portion.

13. The information processing apparatus according to any one of claims 1 to 12, wherein the display screen generation means is configured to generate the display screen such that the additional information is displayed for the target portion, the additional information being specified based on input information that is input.

14. The information processing apparatus according to any one of claims 1 to 13, further comprising display control means configured to control a display unit to display the display screen.

15. An information processing system comprising: the information processing apparatus according to any one of claims 1 to 14; and a display apparatus connected to the information processing apparatus via a network, wherein the information processing apparatus further comprises transmitting means configured to transmit display screen information indicating the display screen to the display apparatus, and the display apparatus includes receiving means configured to receive the display screen information transmitted from the information processing apparatus, and display control means configured to control a display unit to display the display screen.

16. An information processing method comprising: generating a display screen including a spatial image and additional information for a target portion, the spatial image being an image indicating a space including the target portion, the additional information being superimposed on the spatial image, wherein the additional information is generated based on a distance from a position of a point of view from which the spatial image is viewed to the target portion in the spatial image.

17. A recording medium carrying computer readable code for controlling a computer system to carry out an information processing method comprising:generating a display screen including a spatial image and additional information for a target portion, the spatial image being an image indicating a space including the target portion, the additional information being superimposed on the spatial image, wherein the additional information is generated based on a distance from a position of a point of view from which the spatial image is viewed to the target portion in the spatial image.

Images