Input and output apparatus, information processing apparatus, information processing method, program, and information processing system

Abstract

To provide appropriate information to a user who operates a three-dimensional shape.SOLUTION: A terminal apparatus 3 includes a display control unit 34 that displays a three-dimensional shape on a display 308, which is an example of a display unit, and a reception unit 32 that accepts an operation on the three-dimensional shape displayed on the display 308. The display control unit 34 further displays, on the display 308, recommendation information 1700, which is an example of information corresponding to the three-dimensional shape and the operation, based on the operation on the three-dimensional shape displayed on the display 308.SELECTED DRAWING: Figure 16

Description

[Technical field]

[0001] The present invention relates to an input / output device, an information processing device, an information processing method, a program, and an information processing system. [Background technology]

[0002] Patent Document 1 discloses a technique for extracting a matching object to be matched with 3D model data from point cloud data consisting of depth information, matching the 3D model data with the matching object, and identifying object matching points in the 3D model data using machine learning. Patent Document 1 also discloses a technique for identifying the position of an object within a building based on information on the object matching points in the 3D model data and depth information of the point cloud data. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7113611 Summary of the Invention [Problem to be solved by the invention]

[0004] An object of the present invention is to provide appropriate information to a user who manipulates a three-dimensional shape. [Means for solving the problem]

[0005] The input / output device of the present invention comprises a display control means for displaying a three-dimensional shape on a display unit, and a reception means for receiving operations on the three-dimensional shape displayed on the display unit, and the display control means causes the display unit to further display information corresponding to the three-dimensional shape and the operation based on the operation on the three-dimensional shape displayed on the display unit. Effect of the Invention

[0006] According to the present invention, it is possible to provide a user who manipulates a three-dimensional shape with appropriate information corresponding to the three-dimensional shape and the manipulation. [Brief description of the drawings]

[0007] [Figure 1] 1 is an overall configuration diagram of a three-dimensional shape generation system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a hardware configuration diagram of a terminal device and a management server according to the embodiment. [Diagram 3] 1 is a functional block diagram of a three-dimensional shape generation system according to an embodiment of the present invention. [Figure 4] FIG. 4 is a conceptual diagram illustrating an example of a setting information management table according to the embodiment. [Diagram 5] FIG. 11 is a conceptual diagram illustrating an example of a virtual space management table according to the embodiment. [Figure 6] FIG. 11 is a sequence diagram showing an example of a virtual space generation process according to the embodiment. [Figure 7] FIG. 11 is a sequence diagram showing an example of a virtual space operation process according to the embodiment. [Figure 8] FIG. 4 is an explanatory diagram of a setting screen according to the embodiment. [Figure 9] 10 is a flowchart illustrating an example of a three-dimensional shape generation process according to the present embodiment. [Figure 10] 5A to 5C are explanatory diagrams of a registration process according to the embodiment. [Figure 11] 5A to 5C are explanatory diagrams of a noise removal process according to the embodiment. [Figure 12] FIG. 4 is an explanatory diagram of a segmentation process according to the embodiment. [Figure 13] 11 is an explanatory diagram of model matching and replacement processing according to the embodiment. FIG. [Figure 14] FIG. 4 is an explanatory diagram of an operation screen according to the embodiment. [Figure 15] 10 is a flowchart showing an example of a virtual space operation process according to the present embodiment. [Figure 16] FIG. 2 is an explanatory diagram of a virtual space screen according to the embodiment. [Figure 17] FIG. 2 is a second explanatory diagram of the virtual space screen according to the present embodiment. [Figure 18] FIG. 13 is a third explanatory diagram of the virtual space screen according to the present embodiment. [Figure 19] FIG. 4 is a fourth explanatory diagram of the virtual space screen according to the present embodiment. [Figure 20] FIG. 5 is a fifth explanatory diagram of the virtual space screen according to the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] In industries such as civil engineering and construction, the shift to BIM / CIM is being promoted in order to address issues such as the declining birthrate and aging population and to improve labor productivity.

[0009] BIM is an abbreviation for Building Information Modeling, and is a solution that uses a building database that adds attribute data such as cost, finish, and management information to a three-dimensional digital model of a building (hereafter referred to as a 3D model) created on a computer, allowing information to be utilized in every process from building design and construction to maintenance.

[0010] CIM is an abbreviation for Construction Information Modeling, and is a solution for the civil engineering field (general infrastructure such as roads, electricity, gas, and water) proposed following the example of BIM, which was being promoted in the construction field. Like BIM, it is being worked on as a way to improve the efficiency and sophistication of the entire construction production system by sharing information between related parties, mainly using 3D models.

[0011] A key aspect of promoting BIM / CIM is how to easily create 3D models of buildings and public facilities.

[0012] When building new buildings, it is relatively easy to convert to BIM / CIM because the finished product can be modeled from scratch using 3D CAD software. On the other hand, in the case of existing buildings, the hurdles to converting to BIM / CIM are higher because the original blueprints may no longer exist, or may differ from the blueprints due to renovations. Converting existing buildings to BIM in this way is called As-Build BIM, and is an important issue for promoting the future conversion to BIM / CIM.

[0013] One way to realize As-Build BIM is to use a laser scanner (hereafter referred to as LS) to measure the space and create a CAD model from the measured point cloud. Conventionally, the space was measured using photographs and tape measures, and sketched to restore the space, which resulted in a huge amount of work costs, but the introduction of LS has dramatically improved the efficiency of this work.

[0014] While As-Build using LS makes modeling easier, it also creates a task called point cloud processing, which did not exist in conventional work. In general point cloud processing, multi-point measurements are performed using LS, each point cloud is aligned, an integrated point cloud is constructed, unnecessary point clouds such as noise are removed, and finally, the point cloud is converted into a CAD model.

[0015] These processes are performed using the automatic model creation function of commercially available point cloud processing software. However, when different categories of objects, such as pipes, desks, and people, exist in the same space, i.e., in the same point cloud, attempting to generate the three-dimensional shapes of these different categories of objects all at once may result in insufficient accuracy or long processing times.

[0016] On the other hand, generating a three-dimensional shape of an object in a single category requires repeated generation for each required category, which is time-consuming.

[0017] In view of the above problems, the present embodiment aims to appropriately generate a three-dimensional shape with the accuracy required by the user without requiring more processing time and effort than necessary.

[0018] Another issue is how to utilize the constructed BIM / CIM.

[0019] Specifically, 3D space restored by BIM / CIM can be used for design and construction purposes, as well as other work such as maintenance and site inspections. In other words, in addition to using it as a blueprint, other uses such as keeping records in 3D space and sharing it with others can be considered. At this time, 3D contains more information than 2D, making it difficult for first-time viewers to handle.

[0020] In addition, because digital work can be recorded as logs, if tacit knowledge can be extracted from these, it can be effectively used for transferring skills from experienced workers to younger workers. This is expected to lead to front-loading of work and human resource development.

[0021] On the other hand, there are no established standards for how to provide support or present information, so it is important to create a system that makes it easy to utilize 3D space.

[0022] Another object of this embodiment is to prevent oversight by referring to the gaze information of the worker and the search results of the knowledge base, and displaying on the screen the unwatched parts related to the searched knowledge. This allows for the transfer of skills, since the knowledge base records information collected from past experts.

[0023] In this embodiment, when a user performs an operation in a 3D space, information related to the operation is recommended by the following two methods.

[0024] [Recommendation method that does not require learning operations] Predefined information or information extracted by analyzing past work logs is assigned to objects and positions that exist in 3D space in advance, and the above information is recommended when a user performs operations such as looking at, clicking, or touching a specific object or position.

[0025] [Recommendation method that learns how to operate] When operating in 3D space, the user selects from the information presented. The selection history is recorded in a knowledge base and learned from to extract the user's preferences. Then, when an operation is performed in the same space, related information is extracted from the knowledge base and recommended to the user operating the space. For example, information that is easy for the user to select, or information that the user is likely to prefer based on the selection history, is given priority in the recommendation.

[0026] In order to make it easier to use the 3D space, it is desirable to set various conditions such as recommended information, timing of presentation, etc. The following describes the conditions related to the work that are common to all the embodiments.

[0027] Information is recommended to the user by combining the various conditions shown below. Note that by setting each of the various conditions, it is possible to make more detailed recommendations and reduce unnecessary information such as noise. However, it is not necessary to set all of the various conditions, and it is possible to set any items as appropriate.

[0028] 1. How to specify the position When operating in 3D space, there is a method of linking the recommended information with the recommended position. For example, there is a method of linking information with absolute coordinates in the space. When the user performs one of the "operations" described later on this absolute position, the linked information is recommended.

[0029] Other examples of position designation methods include a method of directly assigning information to an object in space, and a method of designating based on the relative positional relationship between objects. The latter is an effective designation method when assigning information based on the relationship between objects, for example, by assigning information A to the midpoint between a table and a chair.

[0030] 2. Timing of recommendation Even for the same space, it is desirable to switch the recommendation timing depending on changes over time. For example, time of day or season can be mentioned. Also, time-to-time changes in the use of the space can be used. For example, in the case of a work process in the construction industry, the situation of the space and the work performed in the space change from an initial site survey, to a detailed site survey, design, construction, and maintenance.

[0031] 3. Eligibility for recommendation This item focuses on the users who use the space. It is assumed that the information required will change if the user changes, even if the space is the same.

[0032] For example, personal information of a user, such as the user's name, age, date of birth, sex, address, affiliation, email address, etc. In the case of construction, the user's role using the space, such as site supervisor or worker, and the qualifications, such as CEO, director, executive officer, department manager, section manager, assistant manager, or regular employee, are examples.

[0033] Other possible divisions include stakeholders such as investors, salespeople, and customers, and industries such as construction, logistics, real estate, entertainment, retail, transportation, information and communications, commerce, finance, services, fisheries, agriculture, forestry, mining, electricity, and gas.

[0034] 4. Media This item describes the media information to be recommended. It is most common to recommend information in text format, but it is also possible to recommend other media. For example, when a user is focusing on a particular object, it is possible to recommend inspection reports (PDF) and related document files related to that object.

[0035] It is also possible to read text aloud or play pre-recorded audio. Also, if you imagine a scene where you want to place a 3D model when using the space, it is possible to present multiple 3D model candidates and select one to place in the space, making 3D models the recommended media.

[0036] 5.Operation This is an item related to the trigger for recommending media. In other words, it corresponds to the part that corresponds to the user's operation, and the media is decided based on this information as input.

[0037] For example, when manipulating a space on a PC, input operations include clicking with input devices such as a mouse or keyboard, moving the mouse, gazing (for example, defining the center of the screen as the user's rest area), viewing operations such as zooming in and out, text input, voice input via a microphone, and file attachments.

[0038] That is, it is possible to recommend information when an arbitrary object is clicked, or to recommend related information when a lot of mouse movement is assumed to mean the user is worried about something. In addition, immersive operations in 3D space, as typified by the metaverse, are also possible. That is, the user becomes an avatar and enters the 3D space, and actions equivalent to the user's behavior in the real space, such as what they are looking at (gaze information) and what they are touching (touch), can be used as triggers as they are. In particular, operations specific to 3D are illustrated in the following embodiment.

[0039] 6. Interface These include non-immersive devices such as PCs, smartphones, and tablets, and immersive devices such as VR goggles and AR glasses.

[0040] As described above, by defining each item in the conditions and its contents in detail, it becomes possible to recommend useful information with less noise.

[0041] It is possible to have the recommendation function be switched on and off from within the system. For example, if the "recommendation function" checkbox is selected, the recommendation function can be switched on.

[0042] 1 is a diagram showing the overall configuration of a three-dimensional shape generating system according to an embodiment of the present invention. The three-dimensional shape processing system 1 of this embodiment is constructed by a terminal device 3, which is an example of an input / output device, and a server 5.

[0043] The server 5 is an example of an information processing device connected to an input / output device via a network. The server 5 is also an example of a three-dimensional shape generating device that generates three-dimensional shape information indicating a three-dimensional shape corresponding to a three-dimensional point group by using model shape information indicating a three-dimensional model shape.

[0044] Here, a three-dimensional point cloud is a collection of coordinate points in a virtual three-dimensional space that can be handled by a computer or the like. A three-dimensional point cloud is also called a point cloud. A three-dimensional point cloud is a collection of coordinate points that correspond to measurement points on the surface of an object when a space containing the object is measured using a laser scanner LS or the like. In addition, color information may be added to each coordinate point, and the RGB value of each coordinate point may be added as color information.

[0045] Although the example shows that the 3D point cloud is measured using a laser scanner LS, other optical measurement means or mechanical measurement means may be used. Optical measurement means include a method using a stereo camera or a method using Visual SLAM.

[0046] Furthermore, three-dimensional shape information is information that indicates the three-dimensional shape of an object that can be handled by a computer or the like. Information that indicates a three-dimensional shape is information that can specify the three-dimensional shape geometrically; for example, in the case of a sphere, the coordinates of the center and the radius correspond to information that indicates the three-dimensional shape. When the three-dimensional shape of an object is expressed as a polyhedron (polygon), the coordinate points of each vertex of the polyhedron are an example of information that indicates the three-dimensional shape. In addition, any information that can uniquely define the shape of the object can be used as information that represents the three-dimensional shape.

[0047] The three-dimensional shape information may include information related to the color and material of the object in addition to information indicating the three-dimensional shape of the object.

[0048] The three-dimensional model shape is a model such as a template used to generate three-dimensional shape information from a three-dimensional point group. The model shape information is information indicating the three-dimensional model shape, and one piece of model shape information corresponds to one three-dimensional model shape.

[0049] The model shape information may include information related to the color and material of the three-dimensional model in addition to the three-dimensional model shape. Specifically, the three-dimensional model shape showing a plane may have information called texture, such as the color and pattern of the wall, and information on the material.

[0050] Texture information makes it easy to create CAD models with colors and patterns. Material information can be passed directly to CAD models that can also have material information, eliminating the need for users to make settings.

[0051] Here, "texture" originally refers to the feel and texture of an object's surface, and in three-dimensional shape information, it refers to a pattern or image that is affixed to the surface of a three-dimensional shape in order to express the surface feel of an object.

[0052] In addition, CAD models are generally created using "surfaces," "solids," and "polygons." However, "solids" contain not only the appearance but also information about the contents such as material, making it possible to calculate mass and volume and to express cross-sectional shapes, making them the closest representation of an actual object.

[0053] Furthermore, setting the color, material, brightness, background, etc. of a CAD model is called "rendering," and by rendering, it is possible to create beautiful images like photographs.

[0054] As described above, the three-dimensional shape information includes three-dimensional geometric shapes handled by 3DCAD, BIM, and CIM, or geometric information, mesh objects, polygon information with texture, and the like.

[0055] The terminal device 3 and the server 5 can communicate with each other via a communication network 100. The communication network 100 is constructed by the Internet, a mobile communication network, a LAN (Local Area Network), and the like. The communication network 100 may include not only wired communication but also wireless communication networks such as 3G (3rd Generation), WiMAX (Worldwide Interoperability for Microwave Access), and LTE (Long Term Evolution). The terminal device 3 can communicate with each other using a short-range communication technology such as NFC (Near Field Communication) (registered trademark).

[0056] <Hardware configuration> 2 is a diagram showing the hardware configuration of the terminal device and the management server according to this embodiment. Each piece of hardware configuration of the terminal device 3 is indicated by a reference number in the 300 range. Each piece of hardware configuration of the server 5 is indicated by a 500 piece of hardware in parentheses.

[0057] The terminal device 3 includes a CPU (Central Processing Unit) 301, a ROM (Read Only Memory) 302, a RAM (Random Access Memory) 303, a HD (Hard Disk) 304, a HDD (Hard Disk Drive) 305, a recording medium 306, a media I / F 307, a display 308, a network I / F 309, a keyboard 311, a mouse 312, a CD-RW (Compact Disc-ReWritable) drive 314, and a bus line 310.

[0058] Among these, the CPU 301 controls the operation of the entire terminal device 3. The ROM 302 stores a program used to drive the CPU 301. The RAM 303 is used as a work area for the CPU 301. The HD 304 stores various data such as programs. The HDD 305 controls reading or writing of various data from the HD 304 under the control of the CPU 301. The media I / F 307 controls reading or writing (storing) of data from a recording medium 306 such as a flash memory. The display 308 displays various information such as a cursor, a menu, a window, characters, or an image. The network I / F 309 is an interface for data communication using the communication network 100. The keyboard 311 is a type of input means having a plurality of keys for inputting characters, numbers, various instructions, and the like. The mouse 312 is a type of input means for selecting and executing various instructions, selecting a processing target, moving the cursor, and the like. The CD-RW drive 314 controls reading or writing of various data from a CD-RW 513 as an example of a removable recording medium.

[0059] The server 5 also includes a CPU 501, a ROM 502, a RAM 503, a HD 504, a HDD 505, a recording medium 506, a media I / F 507, a display 508, a network I / F 509, a keyboard 511, a mouse 512, a CD-RW drive 514, and a bus line 510. These have the same configuration as the above-mentioned configurations (CPU 301, ROM 302, RAM 303, HD 304, HDD 305, recording medium 306, media I / F 307, display 308, network I / F 309, keyboard 311, mouse 312, CD-RW drive 314, and bus line 310), and therefore description thereof will be omitted.

[0060] Note that a CD-R drive or the like may be used instead of the CD-RW drive 314 (514). The terminal device 3 and the server 5 may each be constructed by a single computer, or may be constructed by multiple computers in which each unit (function, means, or storage unit) is divided and assigned arbitrarily.

[0061] FIG. 3 is a functional block diagram of the three-dimensional shape generation system according to this embodiment.

[0062] As shown in Fig. 3, the terminal device 3 has a transmission / reception unit 31, a reception unit 32, a display control unit 34, a determination unit 36, and a storage / readout unit 39. Each of these units is a function or a means for performing a function that is realized when any of the components shown in Fig. 2 operates in response to an instruction from the CPU 301 in accordance with a program loaded from the HD 304 onto the RAM 303. The terminal device 3 also has a storage unit 3000 constructed by the RAM 303 and HD 304 shown in Fig. 2.

[0063] (Functional configuration of terminal device) Next, each component of the terminal device 3 will be described.

[0064] The transmission / reception unit 31 is an example of a transmission means, and is realized by instructions from the CPU 301 shown in FIG. 2 and the network I / F 309, and transmits and receives various data (or information) with other terminals, devices or systems via the communication network 100.

[0065] The reception unit 32 is an example of a reception means, and is realized mainly by commands from the CPU 301 shown in FIG. 2, as well as the keyboard 311 and the mouse 312, and receives various inputs from the user.

[0066] The display control unit 34 is an example of a display control means, and is realized by instructions from the CPU 301 shown in FIG. 2, and causes the display 308, which is an example of a display unit, to display various images and screens.

[0067] The determination unit 36 ​​is an example of a determination means, and is realized by instructions from the CPU 301 shown in FIG. 2, and performs various determinations.

[0068] The memory / read unit 39 is an example of a memory control means, and is executed by instructions from the CPU 301 shown in FIG. 2, as well as the HDD 305, the media I / F 307, and the CD-RW drive 314, to store various types of data in the memory unit 3000, the recording medium 306, and the CD-RW 313, and to read various types of data from the memory unit 3000, the recording medium 306, and the CD-RW 313.

[0069] <Management Server Functionality Configuration> The server 5 has a transmission / reception unit 51, a processing unit 53, a determination unit 55, a generation unit 56, a setting unit 57, and a storage / reading unit 59. Each of these units is a function or a means for performing a function that is realized when any of the components shown in Fig. 2 operates in response to an instruction from the CPU 501 in accordance with a program loaded from the HD 504 onto the RAM 503. The server 5 also has a memory unit 5000 constructed by the HD 504 shown in Fig. 2. The memory unit 5000 is an example of a memory means.

[0070] (Management server functional configuration) Next, each component of the server 5 will be described. The server 5 may be configured to distribute each function among multiple computers. Furthermore, the server 5 will be described as a server computer that exists in a cloud environment, but may be a server that exists in an on-premise environment.

[0071] The transmission / reception unit 51 is an example of a transmission means, and is realized by instructions from the CPU 501 shown in FIG. 2 and the network I / F 509, and transmits and receives various data (or information) with other terminals, devices or systems via the communication network 100.

[0072] The processing unit 53 is an example of a processing means, and is realized by instructions from the CPU 501 shown in Fig. 2, and performs various processes described below. The processing unit 53 is an example of a three-dimensional information generating means that generates three-dimensional shape information.

[0073] The determination unit 55 is an example of a determination means, and is realized by instructions from the CPU 501 shown in FIG. 2, and performs various determinations described later.

[0074] The generating unit 56 is an example of a generating means, and is realized by instructions from the CPU 501 shown in FIG. 2, and generates various screens.

[0075] The setting unit 57 is an example of a setting means, and is realized by commands from the CPU 501 shown in FIG. 2, and performs various settings and determinations, which will be described later.

[0076] 2, and the HDD 505, the media I / F 507, and the CD-RW drive 514, and performs processes such as storing various data in the storage unit 5000, the recording medium 506, and the CD-RW 513, and reading various data from the storage unit 5000, the recording medium 506, and the CD-RW 513. The storage unit 5000, the recording medium 506, and the CD-RW 513 are examples of storage means.

[0077] The memory unit 5000 contains a setting information management DB 5001 composed of a setting information management table, a memory processing management DB 5002, a point cloud management DB 5003, a three-dimensional shape management DB 5004, a model shape management DB 5005, and a virtual space management DB 5006 composed of a virtual space management table.

[0078] The setting information management DB 5001 stores and manages various information, the storage processing management DB 5002 stores and manages various processing programs for generating three-dimensional shapes, the point cloud management DB 5003 stores and manages three-dimensional point cloud information for generating three-dimensional shapes, and the three-dimensional shape management DB 5004 stores and manages three-dimensional shape information generated using model shape information. The model shape management DB 5005 stores and manages model shape information indicating three-dimensional model shapes for each model identification information, and the virtual space management DB 5006 stores and manages information on virtual spaces including three-dimensional shapes.

[0079] FIG. 4 is a conceptual diagram showing an example of a setting information management table according to the present embodiment.

[0080] The setting information management table is a table for managing three-dimensional point cloud data for generating a three-dimensional shape and a history of a process for generating the three-dimensional shape. A setting information management DB 5001 configured by a setting information management table as shown in Fig. 4 is constructed in the storage unit 5000. In this setting information management table, the file name of the three-dimensional point cloud data, the history of the process for generating the three-dimensional shape, and attribute information indicating the attributes of the user are associated and managed for each user ID.

[0081] FIG. 5 is a conceptual diagram showing an example of a virtual space management table according to the present embodiment.

[0082] The virtual space management table is a table for managing information about the virtual space including three-dimensional shape information. A virtual space management DB 5006 is constructed in the storage unit 5000, which is configured by the virtual space management table as shown in FIG.

[0083] In this virtual space management table, model identification information, location information, item information, usage information, attribute information, operation name, operation information, and recommendation information are associated with and managed for each virtual space identification information that identifies a virtual space.

[0084] The model identification information is information for identifying model shape information that indicates the three-dimensional model shape, and the position information is information that indicates the position of the three-dimensional model shape in the virtual space using three-dimensional coordinates of XYZ.

[0085] The article information is information about an article represented by a specific three-dimensional shape composed of a single or multiple model shapes.

[0086] The usage information is information for identifying a usage scene in which the three-dimensional shape displayed on the display 308 is to be used, and includes the time and business in which the three-dimensional shape is to be used.

[0087] The attribute information is information for identifying the attributes of the user who operates the three-dimensional shape displayed on the display 308 .

[0088] The operation type indicates the type of operation for a specific three-dimensional shape. The operation information indicates the operation for a specific three-dimensional shape by an operation specified by three-dimensional coordinates of XYZ. For example, a movement operation is specified by an operation of moving linearly from the three-dimensional coordinates (X1, Y1, Z1) of the start point to the three-dimensional coordinates (X2, Y2, Z2) of the end point, and a rotation operation is specified by an operation of moving in a circle from the three-dimensional coordinates (Xa, Ya, Za) of the start point to the three-dimensional coordinates (Xb, Yb, Zb) of the end point.

[0089] The recommended information is an example of specific information corresponding to a specific three-dimensional shape and a specific operation, and indicates information that may be necessary for a user performing a specific operation on a specific three-dimensional shape.

[0090] The recommended information may include a plurality of pieces of selectable information corresponding to a specific three-dimensional shape and a specific operation, and the information to be presented may be determined based on a priority set in advance.

[0091] In addition, the recommendation information may be specific information corresponding to a plurality of specific three-dimensional shapes and a specific operation on the plurality of specific three-dimensional shapes, or may be specific information corresponding to a specific three-dimensional shape and a plurality of specific operations on the specific three-dimensional shapes.

[0092] When the recommended information is specific information corresponding to a plurality of specific three-dimensional shapes and specific operations on the plurality of specific three-dimensional shapes, the operation information includes a chronological order of the operations, and the recommended information is identified to reflect the chronological order of the operations.

[0093] For example, when taking a measurement, you would click on a pipe A to take a measurement, and then click on the machine body B connected to it to take a measurement.

[0094] In this case, if the user clicks on pipe A and performs any other operation such as measurement, and then clicks on the machine body B, a recommendation such as "Please check pipe F as it has a similar shape" will be displayed.

[0095] This can be achieved by recording operations for a specified period of time as operation information, and when a certain operation is performed, judging whether there is an operation that has a chronological relationship with this operation within this specified period of time. In particular, in maintenance and safety checks, the items to be checked and the order in which they are checked are often fixed, and it is inefficient to carry them out while referring to a manual at hand. Therefore, by utilizing the chronological nature of such operations, it is possible to recommend information such as the next task to be performed.

[0096] The recommended information may be not only predefined, but also determined by data analysis. For example, a method is conceivable in which important information is extracted from raw text (memo) added by a user to a certain point or object in space by using a text mining method such as clustering, and this information is linked (added) to the point or object.

[0097] Another possible method would be to analyze a document file attached to a certain point in the space using information processing technology such as OCR and use the extracted text. For example, if a report summarizing fault information has been attached to a certain machine (fault repair report), the cause of the fault, the fault location, the solution, and the solution procedure could be extracted based on the information extracted from that. Then, similar machines could be searched for using 3D object detection (object search) technology and the above information could be added to them. In addition to intentionally adding information in advance like this, it is also possible to use information extracted from work logs remaining while utilizing the space as recommended information.

[0098] [Manage different information for different operations on the same 3D shape] The virtual space management table described above manages a first operation and first information in association with each other for a first three-dimensional shape, and also manages a second operation and second information in association with each other.

[0099] Specifically, for a three-dimensional shape in which the item information is the floor, operations related to item placement are associated with recommended information related to the comments "The wind will no longer reach private seat A" and "It may not pass through the opening," and are managed in association with each other; operations related to item placement + size change are associated with recommended information related to "Delivery cost after size change" and the comment "It may not pass through the opening," and are managed in association with each other.

[0100] [Manage different information for different 3D shapes for the same operation] Moreover, the virtual space management table manages a first operation on the first three-dimensional shape in association with the first information, and also manages a first operation on the second three-dimensional shape in association with the second information.

[0101] Specifically, the virtual space management table manages operations related to orbiting a three-dimensional shape whose item information is a cylinder in association with recommended information related to "diameter measurement information", and also manages operations related to orbiting a three-dimensional shape whose item information is a cardboard in association with recommended information related to "counting".

[0102] [Manage different information when 3D shapes and operations are different] The virtual space management table manages a first operation on the first three-dimensional shape in association with the first information, and also manages a second operation on the second three-dimensional shape in association with the second information.

[0103] Specifically, the virtual space management table manages operations relating to movement around a three-dimensional shape in which the item information is equipment in association with recommended information relating to "possible movement routes," and also manages operations relating to circumnavigating a three-dimensional shape in which the item information is cardboard in association with recommended information relating to "counting."

[0104] [Manage different information for different conditions for the same 3D shape and the same operation] Furthermore, the virtual space management table manages the first information corresponding to the first condition in association with the first three-dimensional shape and the first operation, and manages the first information corresponding to the second condition in association with the first three-dimensional shape and the first operation.

[0105] [Manage different information for each business condition that uses 3D shapes] Specifically, the virtual space management table manages the item information by associating the task related to the usage information being "design" with the recommendation information related to the comment "Private seat A will no longer be exposed to wind" for the three-dimensional shape of the floor and operations related to item placement, and also by associating the task related to the usage information being "site survey (loading and unloading plan)" with the recommendation information related to "loading and unloading cost."

[0106] Similarly, the virtual space management table manages the operation related to the three-dimensional shape and circulation of the item information which is a pipe by associating the task related to the usage information with "site inspection (estimate preparation)" and recommended information related to "diameter measurement information", and also manages the task related to the usage information with "maintenance" and recommended information related to "degree of loss, risks arising when a loss occurs".

[0107] [Manage different information depending on the time of year using 3D shapes] Furthermore, the virtual space management table manages the item information in relation to the three-dimensional shape of an outdoor air conditioning unit and the operation related to clicking, by associating the season related to the usage information with "summer" and recommended information related to "temperature", and also manages the season related to the usage information with "winter" and recommended information related to "degree of freezing".

[0108] [Manage different information for each condition related to user attributes] In addition, the virtual space management table manages the item information in relation to the three-dimensional shape of the heat source machine and operations related to clicking, by associating the attribute information of "on-site worker" with recommended information related to "inspection points", and also manages the attribute information of "management officer" with recommended information related to "cost of replacement (expenses, time)".

[0109] [Manage information for specific conditions for the same 3D shape and the same operation] Here, the virtual space management table manages the first information corresponding to the first condition in association with the first three-dimensional shape and the first operation, but does not necessarily need to manage the second information corresponding to the second condition in association with the first information.

[0110] [Manage information on specific conditions related to work using 3D shapes] Specifically, the virtual space management table manages the item information, which is the three-dimensional shape of a heat source machine and an operation related to pinching (moving), in association with the fact that the task related to the usage information is "site inspection (loading and unloading plan)" and recommended information related to "spatial highlighting of movable areas." However, if the task related to the usage information is "maintenance management," it does not need to be moved, as this is not equipment replacement, and therefore does not manage the corresponding recommended information.

[0111] Similarly, in the case where the item information is a three-dimensional shape of a ceiling-embedded light and an operation related to clicking is performed, if the task related to the usage information is "site survey (loading and unloading plan)", the virtual space management table does not manage the corresponding recommended information since the light embedded in the ceiling is not an obstacle or cannot be moved because it is embedded.

[0112] [Manage information on specific conditions related to the time when 3D shapes are used] In addition, the virtual space management table manages the item information in relation to the three-dimensional shape of outdoor rebar and the operation related to clicking, in association with the season related to the usage information being "summer" and recommended information related to the "temperature of the main body." However, if the season related to the usage information is "winter," there is no risk of burns even if the temperature becomes high, so the corresponding recommended information is not managed.

[0113] Similarly, in the case where the item information is a three-dimensional shape of a roof and an operation related to clicking is performed, if the season related to the usage information is "summer," the virtual space management table does not manage corresponding recommended information because there is no possibility of the roof collapsing due to snow accumulation.

[0114] [Manage information on specific conditions related to user attributes] Furthermore, the virtual space management table manages the attribute information of "field worker" and the recommendation information of "highlight destination" for the item information of a three-dimensional shape of a desk and the operation of pinching (moving), but if the attribute information is "supervisor," the user can determine the location of the object by himself, so the corresponding recommendation information is not managed.

[0115] Similarly, in the virtual space management table, when the item information is a three-dimensional shape that is a desk and the attribute information is “field worker,” the worker does not need to know who is managing the desk, so the virtual space management table does not manage the corresponding recommended information.

[0116] FIG. 6 is a sequence diagram showing an example of a three-dimensional shape generation process according to this embodiment.

[0117] The reception unit 32 of the terminal device 3 receives an input operation related to user information by a user on the input / output screen displayed on the display 308 (step S1). The transmission / reception unit 31 transmits a request for a setting screen including the user information received in step S1 to the server 5, and the transmission / reception unit 51 of the server 5 receives the request transmitted from the terminal device 3 (step S2).

[0118] Next, the memory / read unit 59 of the server 5 searches the setting information management DB 5001 using the user information included in the request received in step S2 as a search key, to read out the file name of the three-dimensional point cloud data associated with the user information included in the request and the history of the process that generated the three-dimensional shape, and the generation unit 56 of the server 5 generates a setting screen based on the file name and history read by the memory / read unit 59 (step S3).

[0119] This setting screen includes a model information setting screen that accepts model setting operations for setting multiple pieces of model shape information used to generate three-dimensional shape information from multiple pieces of model shape information, a point cloud setting screen that accepts point cloud setting operations for setting a three-dimensional point cloud used to generate three-dimensional shape information, and a processing setting screen that accepts processing setting operations for setting a processing program for generating three-dimensional shape information.

[0120] The processing setting screen includes a display that is initially set based on the history read by the memory / read unit 59 .

[0121] The transmitting / receiving unit 51 transmits setting screen information relating to the setting screen generated in step S3 to the terminal device 3, and the transmitting / receiving unit 31 of the terminal device 3 receives the setting screen information transmitted from the server 5 (step S4). Step S4 is an example of a transmitting step of transmitting model information setting screen information relating to the model information setting screen, and the transmitting / receiving unit 31 is an example of a transmitting means.

[0122] Next, the display control unit 34 of the terminal device 3 causes the display 308 to display the setting screen received in step S4 (step S5). The reception unit 32 of the terminal device 3 receives a predetermined input operation by the user on the displayed setting screen. This input operation includes a model setting operation for setting multiple pieces of model shape information used to generate three-dimensional shape information from multiple pieces of model shape information, a point cloud setting operation, and a process setting operation. Step S5 is an example of a reception step for receiving a model setting operation for setting multiple pieces of model shape information used to generate three-dimensional shape information from multiple pieces of model shape information.

[0123] The transmitting / receiving unit 31 transmits input information related to the input operation received by the receiving unit 32 to the server 5, and the transmitting / receiving unit 51 of the server 5 receives the input information transmitted from the terminal device 3 (step S6). This input information includes model setting information for setting a plurality of pieces of model shape information used to generate three-dimensional shape information from a plurality of pieces of model shape information, point cloud setting information, and processing setting information.

[0124] The memory / read unit 59 of the server 5 updates the history of the process that generated the three-dimensional shape, which is associated with the user information and stored in the setting information management DB 5001, based on the processing setting information and model setting information contained in the input information received in step S6 (step S7).

[0125] Next, the memory / read unit 59 of the server 5 searches the point cloud management DB 5003 using the point cloud setting information included in the input information received in step S6 as a search key, and reads out the three-dimensional point cloud data associated with the point cloud setting information.

[0126] Furthermore, the storage / reading unit 59 searches the stored processing management DB 5002 using the processing setting information included in the input information received in step S6 as a search key, thereby reading out the processing program associated with the processing setting information.

[0127] Then, the storage / reading unit 59 searches the model shape management DB 5005 using the model identification information corresponding to the model setting information included in the input information received in step S6 as a search key, thereby reading out the three-dimensional model shape associated with the model identification information.

[0128] The processing unit 53 of the server 5 generates three-dimensional shape information based on the three-dimensional point cloud data, the processing program, and the three-dimensional model shape read from the storage / read unit 59 (step S8).

[0129] Step S8 is an example of a three-dimensional information generation step in which three-dimensional shape information is generated using a plurality of model shape information set by the model setting operation based on a model setting operation on the model information setting screen 1230 and point cloud information indicating a three-dimensional point cloud.

[0130] The generating unit 56 of the server 5 generates an operation screen for receiving an operation input for generating three-dimensional shape information, and the transmitting / receiving unit 51 transmits operation screen information relating to the operation screen to the terminal device 3 (step S9).

[0131] The transmitter / receiver 31 of the terminal device 3 receives the operation screen information transmitted from the server 5, the display control unit 34 of the terminal device 3 displays the received operation screen on the display 308, and the reception unit 32 of the terminal device 3 receives a predetermined input operation by the user on the displayed operation screen (step S10).

[0132] This input operation includes a shape setting operation for setting one piece of three-dimensional shape information from a plurality of pieces of three-dimensional shape information.

[0133] The transmitting / receiving unit 31 transmits input information relating to the input operation accepted by the accepting unit 32 to the server 5, and the transmitting / receiving unit 51 of the server 5 receives the input information transmitted from the terminal device 3 (step S11).

[0134] This input information includes shape setting information obtained by a shape setting operation, and the processing unit 53 of the server 5 determines three-dimensional shape information based on the shape setting information included in the input information received in step S11.

[0135] The terminal device 3 and the server 5 repeatedly execute steps S8 to S11 as necessary.

[0136] The processing unit 53 converts the generated three-dimensional shape information into a CAD format or the like, and the storage / reading unit 59 stores the converted three-dimensional shape information in the three-dimensional shape management DB 5004, the recording medium 506, or the CD-RW 513 in association with the model identification information of each of the single or multiple model shapes used to generate the three-dimensional shape and the position information in the three-dimensional point cloud data read in step S8 (step S12). The CAD format is three-dimensional shape information that can be handled by 3D CAD. When the three-dimensional shape information is to be handled by a commercially available 3D CAD, the three-dimensional shape information is converted into the 3D CAD format.

[0137] The storage / reading unit 59 updates the history of the process that generated the three-dimensional shape, which is associated with the user information and stored in the setting information management DB 5001, based on the model shape information corresponding to the determined three-dimensional shape information (step S13).

[0138] Next, the storage / readout unit 59 of the server 5 reads out all the three-dimensional shape information generated based on the three-dimensional point cloud data read out in step S8 from the three-dimensional shape management DB 5004 together with the model identification information and position information, and the generation unit 56 of the server 5 generates a virtual space screen including the three-dimensional shapes read out by the storage / readout unit 59 (step S14). This virtual space screen is a screen that shows the space related to the three-dimensional point cloud data read out in step S8 as a virtual space including one or more three-dimensional shapes generated based on the three-dimensional point cloud data.

[0139] The transmitting / receiving unit 51 transmits the virtual space screen information indicating the virtual space screen generated in step S14 to the terminal device 3 (step S15).

[0140] The transmitter / receiver 31 of the terminal device 3 receives the virtual space screen information transmitted from the server 5, and the display control unit 34 of the terminal device 3 causes the display 308 to display the virtual space screen indicated by the received virtual space screen information (step S16).

[0141] The accepting unit 32 of the terminal device 3 accepts a predetermined input operation by the user regarding the displayed virtual space screen. The input operation includes an operation for setting the virtual space identification information, the item information, the usage information, the attribute information, the operation type, the operation information, and the recommendation information shown in FIG.

[0142] The transmitting / receiving unit 31 transmits input information related to the input operation accepted by the accepting unit 32 to the server 5, and the transmitting / receiving unit 51 of the server 5 receives the input information transmitted from the terminal device 3 (step S17). This input information includes virtual space identification information, item information, usage information, attribute information, operation type, operation information, and recommendation information.

[0143] The storage / reading unit 59 of the server 5 associates the virtual space identification information, item information, use information, attribute information, operation type, operation information, and recommendation information contained in the input information received in step S17 with the model identification information and position information read in step S14, and stores them in the virtual space management DB (step S18). As a result, the various information shown in FIG. 5 is stored in the virtual space management DB 5006.

[0144] In the above, the functions of the server 5 in FIG. 3 may be integrated into the terminal device 3, and the processes of the server 5 in FIG.

[0145] FIG. 7 is a sequence diagram showing an example of the virtual space operation process according to this embodiment.

[0146] The accepting unit 32 of the terminal device 3 accepts an input operation related to virtual space identification information on the input / output screen displayed on the display 308 (step S101).

[0147] The transmission / reception unit 31 transmits to the server 5 virtual space identification information related to the input operation accepted by the acceptance unit 32 in step S101, and the transmission / reception unit 51 of the server 5 receives the virtual space identification information transmitted from the terminal device 3 (step S102).

[0148] The memory / read unit 59 of the server 5 searches the virtual space management DB 5006 using the virtual space identification information received in step S102 as a search key, thereby reading out the model identification information, position information and item information associated with the virtual space identification information, and also searches the model shape management DB 5005 using the model identification information read from the virtual space management DB 5006 as a search key, thereby reading out the three-dimensional model shape associated with the model identification information.

[0149] The generation unit 56 of the server 5 generates a virtual space screen based on the model identification information, the three-dimensional model shape, the position information, and the item information read by the storage / reading unit 59 (step S1033).

[0150] The transmitting / receiving unit 51 transmits virtual space screen information indicating the virtual space screen generated in step S103 to the terminal device 3 (step S104).

[0151] The transmitter / receiver 31 of the terminal device 3 receives the virtual space screen information transmitted from the server 5, and the display control unit 34 of the terminal device 3 causes the display 308 to display the virtual space screen indicated by the received virtual space screen information (step S105).

[0152] The accepting unit 32 of the terminal device 3 accepts a predetermined input operation by the user regarding the displayed virtual space screen and an operation regarding a three-dimensional shape included in the virtual space screen. The input operation includes an operation of inputting a user ID and usage information.

[0153] The transmission / reception unit 31 transmits to the server 5 the user ID and usage information related to the input operation received by the reception unit 32, and the item information and operation information related to the operation on the three-dimensional shape, and the transmission / reception unit 51 of the server 5 receives the user ID, usage information, item information and operation information transmitted from the terminal device 3 (step S106).

[0154] The storage / reading unit 59 of the server 5 searches the setting information management DB 5001 using the user ID received in step S106 as a search key, thereby reading out attribute information associated with the user ID, and searches the model shape management DB 5005 using the item information, operation information, and usage information received in step S106 and the attribute information read from the setting information management DB 5001 as search keys, thereby reading out recommended information associated with the item information, operation information, usage information, and attribute information.

[0155] The generating unit 56 of the server 5 generates a recommended information screen including the recommended information read by the storing / reading unit 59 (step S107).

[0156] The transmitting / receiving unit 51 transmits the recommended information screen information indicating the recommended information screen generated in step S107 to the terminal device 3 (step S108).

[0157] The transmitter / receiver 31 of the terminal device 3 receives the recommended information screen information transmitted from the server 5, and the display control unit 34 of the terminal device 3 causes the display 308 to display the recommended information screen indicated by the received recommended information screen information (step S109).

[0158] The accepting unit 32 of the terminal device 3 accepts a user's operation related to the displayed recommended information screen.

[0159] The transmitter / receiver 31 transmits to the server 5 operation information indicating the operation related to the recommended information screen accepted by the accepting unit 32, and the transmitter / receiver 51 of the server 5 receives the operation information transmitted from the terminal device 3 (step S110).

[0160] The storage / reading unit 59 of the server 5 updates the virtual space management table of the virtual space management DB 5006 based on the operation information received in step S110 (step S111).

[0161] In the above, the functions of the server 5 in FIG. 3 may be integrated into the terminal device 3, and the processes of the server 5 in FIG.

[0162] FIG. 8 is an explanatory diagram of a display screen according to this embodiment.

[0163] FIG. 8 shows a display screen 1000 displayed on the display 308 of the terminal device 3 in step S5 of the sequence diagram shown in FIG.

[0164] The display control unit 34 of the terminal device 3 causes the display screen 1000 to display a user information display screen 1100, a setting screen 1200, and a confirmation button 1300, which is an example of an instruction acceptance screen.

[0165] The setting screen 1200 includes a point cloud setting screen 1210, a processing setting screen 1220, and a model information setting screen 1230 (an example of a reception screen).

[0166] The point cloud setting screen 1210 is a screen for receiving a point cloud setting operation for setting point cloud information indicating a three-dimensional point cloud used for generating three-dimensional shape information, and the display control unit 34 displays point cloud setting boxes 1212, 1214 in association with the file names of the multiple point cloud data read by the storage / reading unit 59. A plurality of point cloud setting boxes 1212, 1214 can be set.

[0167] The processing setting screen 1220 is a screen that accepts processing setting operations for setting a processing program for generating three-dimensional shape information, and the display control unit 34 displays processing setting boxes 1222, 1224, and 1226 in association with the names of a plurality of processes, respectively.

[0168] 8, a process setting box 1222 is for setting a process program related to registration, a process setting box 1224 is for setting a process program related to noise removal, and a process setting box 1226 is for setting a process program related to segmentation. Details of these process programs will be described later. Here, the process setting box 1222 related to registration may be set automatically when multiple point clouds are set on the point cloud setting screen 1210. Also, the process related to segmentation may be always executed, and the process setting box 1226 related to segmentation may be omitted.

[0169] The display control unit 34 initially sets and displays the settings in the process setting boxes 1222, 1224, and 1226 based on the history read by the storage / readout unit 59. If there is no history, a predetermined setting is displayed as the initial setting.

[0170] The model information setting screen 1230 is an example of a reception screen that receives a model setting operation for setting a plurality of pieces of model shape information used to generate three-dimensional shape information from a plurality of pieces of model shape information.

[0171] The model information setting screen 1230 includes model information setting boxes 1231 , 1234 , 1235 , and 1236 , model candidate setting boxes 1232 and 1233 , and a model display screen 1240 .

[0172] The display control unit 34 displays model information setting boxes 1231, 1234, and 1235 in association with the names of the multiple pieces of model information, and displays model candidate setting boxes 1232 and 1233 in association with the names of the multiple model shapes. Here, model information indicates a group of model shape information including one or more pieces of model shape information.

[0173] In FIG. 8, model information setting box 1231 is for setting model information indicating the three-dimensional model shape of piping, model information setting box 1234 is for setting model information indicating the three-dimensional model shape of a desk, and model information setting box 1235 is for setting model information indicating the three-dimensional model shape of a person.

[0174] In addition, the model candidate setting box 1232 is for setting model shape information indicating the three-dimensional model shape of the piping model A, and the model shape setting box 1233 is for setting model shape information indicating the three-dimensional model shape of the piping model B.

[0175] The display control unit 34 displays the model candidates 1242, 1243, or the model shape 1244 on the model display screen 1240 in association with the respective names of the set model information or model candidates. If the model shape information includes color, it may be displayed in color. Here, each of the multiple model candidates includes a three-dimensional model shape having a different shape from each other, and is used to generate multiple pieces of three-dimensional shape information from the same region of the three-dimensional point cloud, and the generated multiple pieces of three-dimensional shape information are determined as one piece of three-dimensional shape information in a subsequent process.

[0176] A model candidate 1242 indicates a plurality of three-dimensional model shapes of a piping model A, a model candidate 1243 indicates a plurality of three-dimensional model shapes of a piping model B, and a model shape 1244 indicates a three-dimensional model shape of a desk.

[0177] When a setting box is pointed to with a pointing device such as the mouse 312, the display control unit 34 of the reception unit 32 of the terminal device 3 displays a check mark in the setting box, and the reception unit 32 accepts the various setting operations. When the confirm button 1300 is operated, the reception unit 32 confirms the various setting operations.

[0178] Then, as described in step S6 of FIG. 5, the transmitting / receiving unit 31 transmits to the server 5 input information including various setting information based on the various setting operations received by the receiving unit 32.

[0179] FIG. 9 is a flowchart showing an example of a three-dimensional shape generation process according to this embodiment, and shows a process corresponding to step S8 in FIG.

[0180] The processing unit 53 of the server 5 acquires the three-dimensional point cloud data read from the storage / reading unit 59 (step S21), and if multiple three-dimensional point cloud data are acquired and the processing program read from the storage / reading unit 59 includes a processing program related to registration, executes registration processing (step S22). The registration processing is processing for converting multiple three-dimensional point clouds into one integrated three-dimensional point cloud.

[0181] When the processing programs read from the storage / readout unit 59 include a processing program related to noise removal, the processing unit 53 executes noise removal processing (step S23). The noise removal processing is processing for removing unnecessary points from the three-dimensional point cloud.

[0182] The setting unit 57 of the server 5 sets the model information based on the model setting information included in the input information received in step S6 of Fig. 5 (step S24). For example, as shown in Fig. 8, when the model setting information includes model information indicating a three-dimensional model shape of a pipe and model information indicating a three-dimensional model shape of a desk, the setting unit 57 first sets the model information indicating the three-dimensional model shape of the pipe.

[0183] The setting unit 57 sets one or more pieces of model shape information based on the model setting information included in the input information received in step S6 in Fig. 5 (step S25). For example, as shown in Fig. 8, when the model setting information includes model shape information indicating the three-dimensional model shape of the piping model A and model shape information indicating the three-dimensional model shape of the piping model B as multiple model candidates, the setting unit 57 first sets the model shape information indicating the three-dimensional model shape of the piping model A.

[0184] If the processing programs read from the storage / read unit 59 include a processing program related to segmentation, the processing unit 53 executes a segmentation process (step S26).

[0185] Segmentation processing is also called point cloud classification processing, and techniques such as machine learning PointTransformer are known. (Note) X.-F. Han, Y.-J. Kuang, and G.-Q. Xiao, ''Point cloud learning with transformer,'' 2021, arXiv:2104.13636.

[0186] The segmentation process is a process of labeling specific points in a three-dimensional point cloud so that they can be distinguished from other point clouds, and may be performed in conjunction with a clustering process that groups close points among the labeled points.

[0187] The processing unit 53 compares the point group labeled in step S26 with each of the multiple three-dimensional model shapes included in the model shape information set in step S25, and replaces the specific area with the three-dimensional model shape that is the closest in shape (step S27). If there is no optimal model shape, the processing unit 53 may adjust the dimensions and shape of the model shape. If there is a possibility that the texture information is different for the same shape, the processing unit 53 may also compare models that are the same shape but have different textures. The processing unit 53 may also omit the segmentation process in step S26 and execute step S27 after step S25.

[0188] The determination unit 55 of the server 5 determines whether there is a point cloud for which replacement has not been processed (step S28), and if there is an unprocessed point cloud, the process returns to step 27, and the processing unit 53 executes a process of replacing a specific area with respect to the unprocessed point cloud by using the three-dimensional model shape for the unprocessed point cloud. If the ratio of the unprocessed point cloud to all point clouds becomes equal to or less than a predetermined value, the determination unit 55 may determine that there is no unprocessed point cloud, that is, that the processing of the point clouds is completed.

[0189] On the other hand, if there is no unprocessed point cloud, the determination unit 55 determines whether there is an unprocessed model candidate (step S29). If there is an unprocessed model candidate, the process returns to step 25, and the setting unit 57 sets model shape information included in the unprocessed model candidate. For example, the setting unit 57 sets model shape information indicating the three-dimensional model shape of the piping model B following model shape information indicating the three-dimensional model shape of the piping model A as multiple model candidates.

[0190] When there are no unprocessed model candidates, the processing unit 53 determines three-dimensional shape information based on the shape setting information included in the input information received in step S11 of Fig. 5 (step S30). For example, the setting unit 57 determines one of the three-dimensional shape information generated based on the model shape information of the piping model A and the three-dimensional shape information generated based on the model shape information of the piping model B as the three-dimensional shape information as the multiple model candidates.

[0191] The determination unit 55 determines whether there is unprocessed model information (step S31), and if there is no unprocessed model information, ends the process.

[0192] If there is unprocessed model information, the process returns to step 24, and the setting unit 57 sets the unprocessed model information. For example, the setting unit 57 sets the model information of the desk following the model information of the piping.

[0193] FIG. 10 is an explanatory diagram of the registration process according to this embodiment.

[0194] FIG. 10( a ) shows a first three-dimensional point cloud 1410 , and FIG. 10( b ) shows a second three-dimensional point cloud 1420 .

[0195] As an example, the processing unit 53 converts the first three-dimensional point cloud 1410 and the second three-dimensional point cloud 1420 into a single integrated three-dimensional point cloud by aligning feature points 1410a contained in the first three-dimensional point cloud 1410 with feature points 1420a contained in the second three-dimensional point cloud 1420.

[0196] 11 is an explanatory diagram of the noise removal process according to this embodiment. The processing unit 53 removes unnecessary points 1440 from a three-dimensional point group 1430.

[0197] FIG. 12 is an explanatory diagram of the segmentation process according to this embodiment.

[0198] The processing unit 53 labels certain points in the three-dimensional point cloud 1430 to form a labeled point cloud 1450 (black circles) so that they can be distinguished from other point clouds.

[0199] FIG. 13 is an explanatory diagram of the model matching and replacement process according to this embodiment.

[0200] 13(a) shows a three-dimensional point cloud 1430, FIG. 13(b) shows a model candidate 1460 for pipe A, and FIG. 13(c) shows a model candidate 1470 for pipe B.

[0201] A model candidate 1460 for the piping A includes a plurality of model shapes 1461, 1462, and 1463, each of which is different in shape, and a model candidate 1470 for the piping B includes a plurality of model shapes 1471, 1472, and 1473, each of which is different in shape.

[0202] The processing unit 53 compares the labeled point cloud 1450 (point cloud of black circles) with each of a plurality of model shapes 1461, 1462, and 1463 of the piping A, and replaces a specific area with the model shape that is the closest in shape. If there is no optimal model shape among the plurality of model shapes 1461, 1462, and 1463, the processing unit 53 may adjust the dimensions and shape of the model shape. Here, as one example, the model shape that is the closest in shape is the model shape that is the smallest in total distance between the model shape and each point of the point cloud.

[0203] In cases where there is a possibility that the texture or material may be different for the same shape, multiple textures and multiple materials may also be collated. For example, the color (RGB) of the point cloud is compared with the color of the texture included in the model shape information, and the closest color is set. Specifically, for each point of the point cloud, the difference between the texture color information (RGB) for the model at the closest location and the RGB value of the point cloud is calculated, and the sum of the absolute values ​​is calculated, but the comparison method is not important.

[0204] Furthermore, the processing unit 53 compares the labeled point cloud 1450 (point cloud of black circles) with each of the multiple model shapes 1471, 1472, and 1473 of the piping B, and replaces a specific area with the model shape that is the closest in shape. If there is no optimal model shape, the processing unit 53 may adjust the dimensions and shape of the model shape. If there is no optimal model shape among the multiple model shapes 1471, 1472, and 1473, the processing unit 53 may adjust the dimensions and shape of the model shape.

[0205] FIG. 14 is an explanatory diagram of the operation screen according to this embodiment.

[0206] FIG. 14 shows a display screen 1000 displayed on the display 308 of the terminal device 3 in step S10 of the sequence diagram shown in FIG.

[0207] The display control unit 34 of the terminal device 3 causes the display screen 1000 to display an operation screen 1500 including a generated shape setting screen 1510 and a generated shape display screen 1560, and a confirmation button 1300.

[0208] The generated shape setting screen 1510 is an example of a second reception screen for receiving a shape setting operation for setting one piece of three-dimensional shape information from a plurality of pieces of three-dimensional shape information, and includes generated shape setting boxes 1512 and 1514.

[0209] The generated shape setting box 1512 is used to set a three-dimensional shape generated based on the model shape information of the piping model A, and the generated shape setting box 1514 is used to set a three-dimensional shape generated based on the model shape information of the piping model B.

[0210] The display control unit 34 causes the generated shapes 1562 and 1564 to be displayed on the generated shape display screen 1560 in association with the names of the multiple pieces of model shape information.

[0211] Generated shape 1562 indicates a three-dimensional shape generated based on model shape information of piping model A, and generated shape 1564 indicates a three-dimensional shape generated based on model shape information of piping model B.

[0212] When the generated shape setting box 1512 or 1514 is pointed to with a pointing device such as the mouse 312, the reception unit 32 of the terminal device 3 receives the shape setting operation, and when the confirmation button 1300 is operated, confirms the shape setting operation.

[0213] Then, as described in step S11 of Fig. 5, the transmitting / receiving unit 31 transmits input information including the shape setting information by the shape setting operation received by the receiving unit 32 to the server 5. Note that, in Fig. 14, either the generated shape 1562 or the generated shape 1564 is selected based on the user's operation, but the processing unit 53 of the server 5 may automatically select, from the generated shape 1562 or the generated shape 1564, the one closer to the labeled point cloud, regardless of the user's operation. Here, the closer generated shape is, as an example, the generated shape with the smallest total distance calculated between the generated shape and each point of the point cloud.

[0214] Fig. 15 is a flowchart showing an example of a virtual space operation process according to this embodiment, and shows processes corresponding to steps S105 to S111 in the sequence diagram shown in Fig. 7. Fig. 15(a) shows the process of the terminal device 3, and Fig. 15(b) shows the process of the server 5.

[0215] In FIG. 15(a), the reception unit 32 of the terminal device 3 receives an operation to input a user ID (step S41), and receives an operation to input usage information (step S42).

[0216] The determination unit 35 determines whether the reception unit 32 has received an operation to select an article having a three-dimensional shape included in the virtual space screen displayed on the display 308 (step S43).

[0217] When an item is selected, the determination unit 5 determines whether the reception unit 32 has received a predetermined operation for the selected item (step S44).

[0218] If there is a specified operation on the selected item, the transmission / reception unit 31 transmits to the server 5 the user ID and usage information received by the reception unit 32, item information indicating the selected item, and operation information indicating the specified operation on the selected item (step S45).

[0219] The determination unit 35 determines whether the transmitting / receiving unit 31 has received the recommended information screen information transmitted from the server 5 (step S46), and if the recommended information screen information has been received, the display control unit 34 causes the display 308 to display the recommended information screen indicated by the received recommended information screen information (step S47).

[0220] The determination unit 35 determines whether the reception unit 32 has received a user operation related to the displayed recommended information screen (step S48), and if the operation has been received, the transmission / reception unit 31 transmits operation information indicating the operation related to the recommended information screen received by the reception unit 32 to the server 5 (step S49). As an example, the reception unit 32 receives an operation to select one piece of recommended information from among a plurality of pieces of recommended information.

[0221] In FIG. 15(b), the determination unit 55 of the server 5 determines whether the transmitting / receiving unit 51 has received the user ID, the use information, the item information, and the operation information transmitted from the terminal device 3 (step S51).

[0222] When received in step S51, the storage / reading unit 59 searches the setting information management DB 5001 using the user ID received in step S106 as a search key to read attribute information associated with the user ID, and searches the model shape management DB 5005 using the item information, operation information, and usage information received in step S106 and the attribute information read from the setting information management DB 5001 as search keys to read recommended information associated with the item information, operation information, usage information, and attribute information. The generation unit 56 generates a recommended information screen including the recommended information read by the storage / reading unit 59 (step S52).

[0223] The transmitting / receiving unit 51 transmits recommended information screen information indicating the recommended information screen generated in step S52 to the terminal device 3 (step S53). The determining unit 35 determines whether the transmitting / receiving unit 51 has received the operation information transmitted from the terminal device 3 (step S54).

[0224] If received in step S54, the storage / reading unit 59 updates the virtual space management table of the virtual space management DB 5006 based on the operation information received in step S54 (step S55). As an example, the storage / reading unit 59 updates the multiple pieces of recommended information stored in the virtual space management DB 5006 to the selected single piece of recommended information based on the operation information indicating an operation of selecting one piece of recommended information from among the multiple pieces of recommended information.

[0225] In the above, the recommended information may not reflect the intention of the user to whom the recommendation is made. In other words, since the recommendation is made based only on the user's operation, the information recommended may not be what the user wants.

[0226] For example, if the user moves the mouse a lot, it may be determined that the user is unsure of what to do, and useful information may be recommended in this case. However, in reality, the user may be moving the mouse a lot in order to check a specific object from various angles. Therefore, if the operation is misinterpreted in this way, the recommended information may become noise, or the recommendation itself may become noise.

[0227] In order to solve such a problem, the reception unit 32 of the terminal device 3 may receive input of text, documents, voice, and the like in addition to the user ID and usage information.

[0228] In this case, the server 5 analyzes the information input by the user within the system, searches for potentially relevant information from a group of pre-assigned information or from a knowledge base extracted by data mining, and recommends to the user information that is likely to be relevant.

[0229] For example, based on information entered such as "There is a crack in this pipe...", the system can predict that the user will want to know whether the crack needs to be repaired, or who to contact in case of damage, and can present a solution to that question.

[0230] Since users have some intention when they input information, it is easier to understand their intention by utilizing the input information. In this way, adding part of the user's intention as input makes it easier to recommend the information the user needs from the countless pieces of information available.

[0231] In the above, it is assumed that both user operation and information input from the user are used as triggers, but in some cases operation may not be used. For example, for a certain location, recommended information may be selected from an infinite number of pieces of information using only information input. User operation serves the function of first narrowing down the infinite number of pieces of information.

[0232] FIG. 16 is an explanatory diagram of the virtual space screen according to this embodiment.

[0233] 16(a), (c), and (e) show a display screen 1000 displayed on the display 308 of the terminal device 3 in step S105 of the sequence diagram shown in FIG. 7, and FIGS. 16(b), (d), and (f) show top views corresponding to FIGS. 16(a), (c), and (e).

[0234] FIG. 16(g) shows a display screen 1000 displayed on the display 308 of the terminal device 3 in step S109 of the sequence diagram shown in FIG. 7, and FIG. 16(h) shows a top view corresponding to FIG. 16(g).

[0235] In FIG. 16(a), the display control unit 34 of the terminal device 3 causes the display screen 1000 to display a virtual space screen 1600 including a cylinder 1620, which is an example of an article having a three-dimensional shape, as seen from a viewpoint LS.

[0236] When the user performs an operation OP to select cylinder 1620 with pointer 1610 , reception unit 32 receives the selection of cylinder 1620 , and the user starts an operation OP on cylinder 1620 with pointer 1610 .

[0237] As shown in FIG. 16(c), when the user performs an operation OP to rotate around the cylinder 1620 using the pointer 1610, the reception unit 32 receives the rotation operation OP, and the display control unit 34 displays on the display screen 1000 a virtual space screen 1600 in which the viewpoint LS is changed in accordance with the rotation operation OP, as shown in FIG. 16(d).

[0238] As shown in FIG. 16(e), when the user continues the operation OP of orbiting the cylinder 1620 with the pointer 1610, the reception unit 32 receives the orbiting operation OP, and the display control unit 34 displays on the display screen 1000 a virtual space screen 1600 in which the viewpoint LS is further changed in accordance with the orbiting operation OP, as shown in FIG. 16(f).

[0239] As shown in FIG. 16(g), when the user further continues the operation OP of circling the cylinder 1620 with the pointer 1610, the accepting unit 32 accepts the circling operation OP.

[0240] Then, as described in step S45 of FIG. 15, transmitting / receiving unit 31 transmits to server 5 the user ID and the usage information, as well as the article information indicating the cylinder and operation information indicating an operation to circle the cylinder.

[0241] Thereafter, as described in step S46 of FIG. 15, when the transmitter / receiver 31 receives the recommended information screen information transmitted from the server 5, the display control unit 34 causes the recommended information screen 1700 including the recommended information to be superimposed on the virtual space screen 1600 and displayed on the display screen 1000.

[0242] In FIG. 16(g), as described in step S52 of FIG. 15, the storage / reading unit 59 of the server 5 reads out from the model shape management DB 5005 the measurement information of the cylinder 1620, such as “This is a cylinder with a circumference of ~ cm and a height of ~ cm,” as recommendation information associated with the article information indicating the cylinder, operation information indicating an operation of circling the cylinder, usage information, and attribute information corresponding to the user ID, and the display control unit 34 displays a recommendation information screen 1700 including the recommendation information on the display screen 1000.

[0243] FIG. 17 is a second explanatory diagram of the virtual space screen according to this embodiment.

[0244] Similar to FIG. 16, FIGS. 17(a), (c), and (e) show the display screen 1000 displayed on the display 308 of the terminal device 3 in step S105 of the sequence diagram shown in FIG. 7, and FIGS. 17(b), (d), and (f) show top views corresponding to FIGS. 17(a), (c), and (e).

[0245] FIG. 17(g) shows a display screen 1000 displayed on the display 308 of the terminal device 3 in step S109 of the sequence diagram shown in FIG. 7, and FIG. 17(h) shows a top view corresponding to FIG. 17(g).

[0246] In FIG. 17(a), the display control unit 34 of the terminal device 3 causes the display screen 1000 to display a virtual space screen 1600 including a plurality of cardboard boxes 1630, which are an example of an article having a three-dimensional shape, and seen from a viewpoint LS.

[0247] When the user performs an operation to surround a plurality of cardboard boxes 1630 with the pointer 1610, the acceptance unit 32 accepts the selection of the plurality of cardboard boxes 1630, and the user starts an operation OP on the plurality of cardboard boxes 1630 with the pointer 1610.

[0248] As shown in FIG. 17(c), when a user performs an operation OP to rotate around multiple cardboard boxes 1630 using a pointer 1610, the reception unit 32 receives the rotation operation OP, and the display control unit 34 displays on the display screen 1000 a virtual space screen 1600 in which the viewpoint LS is changed in accordance with the rotation operation OP, as shown in FIG. 17(d).

[0249] As shown in FIG. 17(e), when the user continues the operation OP of orbiting multiple cardboard boxes 1630 with the pointer 1610, the reception unit 32 receives the orbiting operation OP, and the display control unit 34 displays on the display screen 1000 a virtual space screen 1600 in which the viewpoint LS is further changed in accordance with the orbiting operation OP, as shown in FIG. 17(f).

[0250] As shown in FIG. 17(g), when the user continues the operation OP of moving around the multiple cardboard boxes 1630 with the pointer 1610, the acceptance unit 32 accepts the operation OP of moving around.

[0251] Then, as described in step S45 of FIG. 15, the transmitter / receiver 31 transmits to the server 5, together with the user ID and usage information, item information indicating the multiple cardboard boxes and operation information indicating an operation to move around the multiple cardboard boxes.

[0252] Thereafter, as described in step S46 of FIG. 15, when the transmitter / receiver 31 receives the recommended information screen information transmitted from the server 5, the display control unit 34 causes the recommended information screen 1700 including the recommended information to be superimposed on the virtual space screen 1600 and displayed on the display screen 1000.

[0253] In Figure 17 (g), as explained in step S52 of Figure 15, the memory / reading unit 59 of the server 5 reads out from the model shape management DB 5005 item information indicating multiple cardboard boxes, operation information indicating an operation to move around the multiple cardboard boxes, usage information, and counting information of multiple cardboard boxes 1630, such as "The number of objects within the selected range is ~," as recommended information associated with attribute information corresponding to the user ID, and the display control unit 34 displays a recommended information screen 1700 including the recommended information on the display screen 1000.

[0254] FIG. 18 is a third explanatory diagram of the virtual space screen according to this embodiment.

[0255] 18(a) shows a display screen 1000 displayed on the display 308 of the terminal device 3 in step S105 of the sequence diagram shown in FIG. 7, and FIG. 18(b) shows a display screen 1000 displayed on the display 308 of the terminal device 3 in step S109 of the sequence diagram shown in FIG. 7.

[0256] In FIG. 18(a), the display control unit 34 of the terminal device 3 causes the display screen 1000 to display a virtual space screen 1600 including a floor 1640, which is an example of an article having a three-dimensional shape.

[0257] As shown in FIG. 18(b), when a user uses pointer 1610 to perform an operation OP to place equipment 1650, which is an example of an item having a three-dimensional shape, on floor 1640, reception unit 32 accepts the operation OP to select floor 1640 and place equipment 1650 on floor 1640.

[0258] Then, as described in step S45 of FIG. 15, the transmitter / receiver 31 transmits to the server 5 the user ID and the usage information, as well as item information indicating the floor 1640 and operation information indicating the operation of placing the equipment 1650 on the floor 1640.

[0259] Thereafter, as described in step S46 of FIG. 15, when the transmitter / receiver 31 receives the recommended information screen information transmitted from the server 5, the display control unit 34 causes the recommended information screen 1700 including the recommended information to be superimposed on the virtual space screen 1600 and displayed on the display screen 1000.

[0260] In FIG. 18(b), as explained in step S52 of FIG. 15, the memory / read unit 59 of the server 5 reads out from the model shape management DB 5005 multiple comments such as “Air will not reach private seat A” and “It may not pass through the opening” as recommended information associated with the item information indicating the floor, operation information indicating the operation of placing equipment on the floor, usage information, and attribute information corresponding to the user ID, and the display control unit 34 displays recommended information screens 1700A and 1700B including the recommended information on the display screen 1000.

[0261] Thereafter, as described in the flowchart of FIG. 15, when the receiving unit 32 receives an operation to select, for example, the recommended information screen 1700A from the recommended information screens 1700A and 1700B, the transmitting / receiving unit 31 transmits operation information indicating the operation to select the recommended information screen 1700A to the server 5.

[0262] Then, when the transmission / reception unit 51 of the server 5 receives the operation information transmitted from the terminal device 3, the storage / reading unit 59 updates, based on the received operation information, for example, multiple pieces of recommended information stored in the virtual space management DB 5006 to a single piece of recommended information displayed on the recommended information screen 1700A.

[0263] When multiple pieces of recommended information are displayed as in FIG. 18, the information selected by the user may be recorded and stored in a knowledge base by introducing a mechanism for learning the information.

[0264] This allows content that is of interest to each user to be prioritized and recommended, and allows the system to learn their preferences.

[0265] For example, even if the attribute information shown in Figure 5 is not stored and managed, it will be possible to easily recommend information regarding costs and safety to site supervisors, to easily recommend work procedures to site workers, and to easily recommend information regarding inspection points to maintenance managers.

[0266] The learning / recommendation can be done for each user, or for example, for each group to which the user belongs. It is also possible to use the user's personal information to recommend information similar to the user's gender or age group.

[0267] One possible method for collecting user preferences is to recommend things in a Q&A format, encouraging users to make a selection by clicking. For example, instead of presenting "This is dangerous," it would be possible to introduce a recommendation method such as "There are safety concerns about this. Would you like to display this?"

[0268] In the former case, it is expected that after the information is presented, the user will close the screen without making a selection, but in the latter case, the user will click to find out the content, and at the same time, the system can collect a log of that selection, making it possible to learn the user's preferences without the user being aware of it.

[0269] In addition, recommended items can be prioritized based on the learned preferences of the user. As a method of prioritizing the presentation, low-priority information may be excluded from the presentation in the first place, or high-priority information may be displayed in bold or in a larger font size, as in a word cloud, so that the importance of the information can be visually shown. In the case of a word cloud, only easy-to-understand text may be displayed, and detailed information may be displayed when the user selects a specific text. Also, a hierarchical confirmation method such as a tree diagram or a list display may be used.

[0270] The server 5 is provided with a learning unit, which allows it to record recommended information selected by the user. This selected information is linked to other related information, such as the attribute information and the business included in the usage information, to learn preferences. This learning makes it possible to recommend information that is predicted to be highly relevant to the same person or users with the same attributes.

[0271] FIG. 19 is a fourth explanatory diagram of the virtual space screen according to this embodiment.

[0272] FIG. 19 shows a display screen 1000 displayed on the display 308 of the terminal device 3 in step S109 of the sequence diagram shown in FIG.

[0273] In FIG. 19, the display control unit 34 of the terminal device 3 causes the display screen 1000 to display a virtual space screen 1600 including a floor 1640, which is an example of an article having a three-dimensional shape.

[0274] When the user uses pointer 1610 to place equipment 1650, which is an example of an item having a three-dimensional shape, on floor 1640 and performs an operation OP to change the size of equipment 1650, reception unit 32 accepts the selection of floor 1640 and the operation OP to place equipment 1650 on floor 1640 and change the size.

[0275] Then, as described in step S45 of FIG. 15, the transmitter / receiver 31 transmits to the server 5, together with the user ID and usage information, item information indicating the floor 1640 and operation information indicating an operation to place and resize the equipment 1650 on the floor 1640.

[0276] Thereafter, as described in step S46 of FIG. 15, when the transmitter / receiver 31 receives the recommended information screen information transmitted from the server 5, the display control unit 34 causes the recommended information screen 1700 including the recommended information to be superimposed on the virtual space screen 1600 and displayed on the display screen 1000.

[0277] As explained in step S52 of FIG. 15, the memory / read unit 59 of the server 5 reads out from the model shape management DB 5005 the item information indicating the floor, the operation information indicating the operation of placing equipment on the floor and changing the size, the usage information, and the comments such as “If the volume = ○ m^3, the delivery cost will be ~ yen” and “At this size, it may not fit through the opening” as recommended information associated with the attribute information corresponding to the user ID, and the display control unit 34 displays the recommended information screen 1700 including the recommended information on the display screen 1000.

[0278] FIG. 20 is a fifth explanatory diagram of the virtual space screen according to this embodiment.

[0279] In FIG. 20, the display control unit 34 of the terminal device 3 causes the display screen 1000 to display a virtual space screen 1600 including a floor 1640 and equipment 1650, which are examples of articles having a three-dimensional shape.

[0280] When the user performs an operation OP of moving the equipment 1650 on the floor 1640 using the pointer 1610, the reception unit 32 receives the selection of the equipment 1650 and the operation OP of moving the equipment 1650.

[0281] Then, as described in step S45 of FIG. 15, the transmitting / receiving unit 31 transmits to the server 5 the user ID and the usage information, as well as the item information indicating the item 1650 and operation information indicating an operation to move the item 1650.

[0282] Thereafter, as described in step S46 of FIG. 15, when the transmitter / receiver 31 receives the recommended information screen information transmitted from the server 5, the display control unit 34 causes the recommended information screens 1660, 1670, and 1700 including the recommended information to be superimposed on the virtual space screen 1600 and displayed on the display screen 1000.

[0283] As explained in step S52 of FIG. 15, the memory / read unit 59 of the server 5 reads out from the model shape management DB 5005 the item information indicating the equipment, the operation information indicating the operation of moving the equipment, the usage information, and the comments such as "The drivable area will be highlighted (surface)" and "The shortest route to point A will be highlighted (lined) in blue (distance: ~m)" as recommended information associated with the attribute information corresponding to the user ID, and the display control unit 34 displays recommended information screens 1660, 1670 and 1700 including the recommended information on the display screen 1000.

[0284] ●Summary● [First aspect] As described above, the terminal device 3, which is an example of an input / output device according to one embodiment of the present invention, includes a display control unit 34 that displays a three-dimensional shape on the display 308, which is an example of a display unit, and a reception unit 32 that receives operations on the three-dimensional shape displayed on the display 308, and the display control unit 34 further displays recommended information 1700, which is an example of specific information corresponding to the specific three-dimensional shape and the specific operation, on the display 308 based on a specific operation on the specific three-dimensional shape displayed on the display 308.

[0285] This makes it possible to provide the user with recommendation information 1700 corresponding to a specific three-dimensional shape and a specific operation.

[0286] [Second aspect] In the first aspect, the display control unit 34 causes the display 308 to display first information corresponding to a first three-dimensional shape and a first operation based on a first operation on a first three-dimensional shape displayed on the display 308, and causes the display 308 to display second information corresponding to the first three-dimensional shape and a second operation based on a second operation on the first three-dimensional shape displayed on the display 308.

[0287] This makes it possible to provide the user with different information for each different operation, even for the same three-dimensional shape.

[0288] [Third aspect] In the first aspect, the display control unit 34 causes the display 308 to display a first three-dimensional shape and first information corresponding to the first operation based on a first operation on a first three-dimensional shape displayed on the display 308, and causes the display 308 to display a second three-dimensional shape and second information corresponding to the first operation based on a first operation on a second three-dimensional shape displayed on the display 308.

[0289] This makes it possible to provide the user with different information for each different three-dimensional shape even when performing the same operation.

[0290] [Fourth aspect] In the first aspect, the display control unit 34 causes the display 308 to display first information corresponding to a first three-dimensional shape and a first operation based on a first operation on a first three-dimensional shape displayed on the display 308, and causes the display 308 to display second information corresponding to a second three-dimensional shape and a second operation based on a second operation on a second three-dimensional shape displayed on the display 308.

[0291] This makes it possible to provide different information to the user when both the three-dimensional shape and the operation are different.

[0292] [Fifth aspect] In the first embodiment, when a specific condition is met based on a specific operation on a specific three-dimensional shape displayed on the display 308, the display control unit 34 causes the display 308 to display recommended information 1700 corresponding to the specific three-dimensional shape, the specific operation, and the specific condition.

[0293] This makes it possible to provide the user with recommendation information 1700 that corresponds to a specific three-dimensional shape, a specific operation, and a specific condition.

[0294] [Sixth aspect] In the first aspect, based on a first operation on a first three-dimensional shape displayed on the display 308, if a first condition is met, the display control unit 34 causes the display 308 to display first information corresponding to the first three-dimensional shape, the first operation, and the first condition, and if a second condition is met, causes the display 308 to display second information corresponding to the first three-dimensional shape, the first operation, and the second condition.

[0295] This makes it possible to provide the user with different information for each condition, even for the same three-dimensional shape and the same operation.

[0296] [Seventh aspect] In the sixth embodiment, the receiving unit 32 further receives an input of predetermined input information, and the first condition and the second condition are conditions related to the input information.

[0297] This makes it possible to provide the user with different information depending on conditions related to input information such as text, document, and voice, even when the same three-dimensional shape and the same operation are used.

[0298] [Eighth aspect] In a seventh aspect, the input information includes user information that identifies a user, or usage information that identifies a usage scene that utilizes the three-dimensional shape displayed on the display 308, and the first condition and the second condition are conditions related to the user information or the usage information.

[0299] This makes it possible to provide the user with different information for each condition related to the user information or usage information, even for the same three-dimensional shape and the same operation.

[0300] [Ninth aspect] In an eighth embodiment, the first condition and the second condition are conditions related to a user's attribute, a time or a business in which the three-dimensional shape is used.

[0301] This makes it possible to provide different information to a user depending on the user's attributes, the time or business in which the three-dimensional shape is used, even when the same three-dimensional shape and the same operation are used.

[0302] [Tenth aspect] In any of the first to ninth aspects, the operation includes an operation specified by three-dimensional coordinates.

[0303] This makes it possible to provide the user with recommendation information 1700 corresponding to a specific three-dimensional shape and a specific three-dimensional operation.

[0304] [Eleventh aspect] In any of the first to tenth aspects, the receiving unit 32 further receives an instruction operation to generate three-dimensional shape information indicating a three-dimensional shape based on point cloud information indicating a three-dimensional point cloud, and the display control unit 34 causes the display 308 to display the three-dimensional shape related to the three-dimensional shape information generated based on the point cloud information.

[0305] This makes it possible to display a three-dimensional shape generated based on a three-dimensional point group, and to provide the user with recommended information 1700 corresponding to a specific three-dimensional shape and a specific three-dimensional operation.

[0306] [12th aspect] The server 5, which is an example of an information processing device according to one embodiment of the present invention, is equipped with a transceiver unit 31, which is an example of a transmitting means for transmitting display screen information indicating a display screen displaying a three-dimensional shape to the terminal device 3, and the transceiver unit 31 receives input information transmitted from the terminal device 3 indicating that a specific operation has been performed on a specific three-dimensional shape displayed on the display screen, and transmits specific information corresponding to the specific three-dimensional shape and the specific operation to the terminal device 3 based on the input information.

[0307] This makes it possible to provide the user of the terminal device 3 with recommended information 1700 corresponding to a specific three-dimensional shape and a specific operation.

[0308] [13th aspect] An information processing method executed by a terminal device 3, which is an example of an information processing device according to one embodiment of the present invention, executes a first display step of displaying a three-dimensional shape on a display 308, a reception step of receiving an operation on the three-dimensional shape displayed on the display 308, and a second display step of displaying a specific three-dimensional shape and specific information corresponding to the specific operation on the display 308 based on a specific operation on the specific three-dimensional shape displayed on the display 308.

[0309] [14th aspect] A program according to one embodiment of the present invention causes a computer to execute a first display step of displaying a three-dimensional shape on the display 308, a reception step of receiving an operation on the three-dimensional shape displayed on the display 308, and a second display step of displaying a specific three-dimensional shape and specific information corresponding to the specific operation on the display 308 based on a specific operation on the specific three-dimensional shape displayed on the display 308.

[0310] [15th aspect] A three-dimensional shape processing system 1, which is an example of an information processing system according to one embodiment of the present invention, comprises a terminal device 3 and a server 5 connected to the terminal device 3 via a network 100. The terminal device 3 comprises a display control unit 34 that displays a three-dimensional shape on a display 308, a reception unit 32 that receives operations on the three-dimensional shape displayed on the display 308, and a transmission / reception unit 31 that transmits input information to the server 5 indicating that a specific operation has been performed on a specific three-dimensional shape displayed on the display 308. The server 5 comprises a transmission / reception unit 51 that receives the input information transmitted from the terminal device 3 and transmits specific information corresponding to the specific three-dimensional shape and the specific operation to the terminal device 3 based on the input information. The transmission / reception unit 31 receives the specific information transmitted from the server 5, and the display control unit 34 displays the specific information on the display 308. [Explanation of symbols]

[0311] 1. Three-dimensional shape processing system 100 Communication Network 3. Terminal device (an example of an input / output device) 31 Transmitting / receiving unit (an example of a transmitting means or a receiving means) 32 Reception unit (an example of a reception means) 34 Display control unit (an example of a display control means) 36 Judgment Department 39 Memory / read unit (an example of a memory control means) 308 Display (an example of a display unit) 3000 storage section 5 Server (an example of a 3D shape generation device) 51 Transmitting / receiving unit (an example of a transmitting means or a receiving means) 53 Processing unit (an example of a three-dimensional information generating means) 55 Judgment Department 56 Generation part 57 Setting section 59 Memory / read unit (an example of a memory control means) 5000 storage section 5001 Setting information management DB (an example of setting information management means) 5002 Memory processing management DB (an example of memory processing management means) 5003 Point cloud management DB (an example of point cloud management means) 5004 Three-dimensional shape management DB (an example of a three-dimensional shape management means) 5005 Model shape management DB 5006 Virtual space management database 1000 display screen 1100 User information display screen 1200 Setting screen 1210 Point Cloud Setting Screen 1212, 1214 Point Cloud Settings Box 1220 Processing setting screen 1222, 1224, 1226 Processing setting box 1230 Model information setting screen (example of reception screen) 1231, 1234, 1235 Model Information Setting Box 1232, 1233 Model candidate setting box 1240 Model Display Screen 1242, 1243 model candidates 1244 Model Shape 1300 Confirm button (example of instruction reception screen) 1410 First 3D point cloud 1420 Second 3D Point Cloud 1430 3D point cloud 1440 Unwanted points 1450 Labeled Point Cloud 1460, 1470 model candidates 1461, 1462, 1463, 1471, 1472, 1473 Model Shape 1500 operation screen 1510 Generation shape setting screen (an example of the second reception screen) 1512, 1514 Generated shape setting box 1560 Generated shape display screen 1562, 1564 Generated shape 1600 Virtual Space Screen 1610 Pointer 1620 3D shape (cylinder) 1630 3D shape (multiple cardboard boxes) 1640 Three-dimensional shape (floor) 1650 3D Shape (Equipment) 1660 Recommended Information (Root) 1670 Recommended Information (Destination) 1700 Recommended Information (Example of Specific Information) OP operation LS Viewpoint

Claims

1. A display control means for displaying a three-dimensional shape on a display unit; a reception means for receiving an operation for the three-dimensional shape displayed on the display unit; Equipped with The display control means An input / output device that causes the display unit to further display, based on the operation on the three-dimensional shape displayed on the display unit, information corresponding to the three-dimensional shape and the operation.

2. The display control means based on a first operation on the first three-dimensional shape displayed on the display unit, displaying first information corresponding to the first three-dimensional shape and the first operation on the display unit; The input / output device according to claim 1 , further comprising: a display unit configured to display second information corresponding to the first three-dimensional shape and the second operation based on a second operation on the first three-dimensional shape displayed on the display unit.

3. The display control means based on a first operation on the first three-dimensional shape displayed on the display unit, displaying first information corresponding to the first three-dimensional shape and the first operation on the display unit; The input / output device according to claim 1 , further comprising: a display unit configured to display, on the display unit, second information corresponding to the second three-dimensional shape and the first operation based on the first operation on the second three-dimensional shape displayed on the display unit.

4. The display control means based on a first operation on the first three-dimensional shape displayed on the display unit, displaying first information corresponding to the first three-dimensional shape and the first operation on the display unit; The input / output device according to claim 1 , further comprising: a display unit configured to display, on the display unit, second information corresponding to the second three-dimensional shape and the second operation based on the second operation on the second three-dimensional shape displayed on the display unit.

5. The display control means Based on the operation on the three-dimensional shape displayed on the display unit, 2. The input / output device according to claim 1, wherein, when a specific condition is met, the display unit displays the three-dimensional shape, the operation, and information corresponding to the specific condition.

6. The display control means Based on the first operation on the first three-dimensional shape displayed on the display unit, when a first condition is satisfied, displaying on the display unit first information corresponding to the first three-dimensional shape, the first operation, and the first condition; The input / output device according to claim 5 , further comprising: a display unit that displays, when a second condition is satisfied, second information corresponding to the first three-dimensional shape, the first operation, and the second condition.

7. The receiving means further receives input of predetermined input information, 7. The input / output device according to claim 6, wherein the first condition and the second condition are conditions related to the input information.

8. the input information includes user information for identifying a user or usage information for identifying a usage scene in which the three-dimensional shape displayed on the display unit is used, 8. The input / output device according to claim 7, wherein the first condition and the second condition are conditions related to the user information or the usage information.

9. 9. The input / output device according to claim 8, wherein the first condition and the second condition are conditions related to a user's attribute, a time or a business in which the three-dimensional shape is to be used.

10. The input / output device according to claim 1 , wherein the operation includes an operation specified by three-dimensional coordinates.

11. The receiving means further receives an instruction operation for instructing to generate three-dimensional shape information indicating a three-dimensional shape based on point cloud information indicating a three-dimensional point cloud; 2. The input / output device according to claim 1, wherein the display control means causes the display unit to display the three-dimensional shape related to the three-dimensional shape information generated based on the point cloud information.

12. a transmitting means for transmitting display screen information indicating a display screen for displaying a three-dimensional shape to an input / output device; a receiving means for receiving input information transmitted from the input / output device, the input information indicating that an operation has been performed on the three-dimensional shape displayed on the display screen; Equipped with The transmitting means is An information processing device that transmits information corresponding to the three-dimensional shape and the operation to the input / output device based on the input information.

13. An information processing method executed by an information processing device, a first display step of displaying the three-dimensional shape on a display unit; a receiving step of receiving an operation for the three-dimensional shape displayed on the display unit; a second display step of displaying, on the display unit, the three-dimensional shape and information corresponding to the operation based on the operation on the three-dimensional shape displayed on the display unit; An information processing method for performing the above.

14. a first display step of displaying the three-dimensional shape on a display unit; a receiving step of receiving an operation for the three-dimensional shape displayed on the display unit; a second display step of displaying, on the display unit, the three-dimensional shape and information corresponding to the operation based on the operation on the three-dimensional shape displayed on the display unit; A program that causes a computer to execute the following.

15. An information processing system including an input / output device and an information processing device connected to the input / output device via a network, The input / output device is A display control means for displaying a three-dimensional shape on a display unit; a reception means for receiving an operation for the three-dimensional shape displayed on the display unit; a transmitting means for transmitting input information indicating that the operation on the three-dimensional shape displayed on the display unit has been performed to the information processing device; Equipped with The information processing device includes: a receiving means for receiving the input information transmitted from the input / output device; a transmitting means for transmitting information corresponding to the three-dimensional shape and the operation to the input / output device based on the input information; Equipped with The input / output device is A receiving unit is further provided for receiving the information transmitted from the information processing device, The display control means An information processing system that displays the information on the display unit.

Images