Real-time communication support system and method, mobile terminal, server, and program

Abstract

The present invention provides a real-time communication assistance system for simultaneously displaying fine 3D point cloud data and small-sized mesh data. Provided is a real-time communication assistance system comprising: a means (11) that photographs a subject; a means (12) that acquires 3D point cloud data from the photographed subject; a means (13) that acquires a landscape mesh screen obtained by converting objects other than the subject into meshes; a means (14) that stores the 3D point cloud data of the subject and the landscape mesh screen; and a means (15) that simultaneously displays the 3D point cloud data of the subject and the landscape mesh screen.

Description

[Technical Field]

[0001] The present disclosure relates to a real-time communication support system, a real-time communication support method, a mobile terminal, a server, and program Regarding. [Background technology]

[0002] In recent years, systems for inspecting, monitoring, maintaining, and operating infrastructure and facilities have been proposed that utilize 3D sensors such as LiDAR (Light Detection and Ranging) and ToF (Time of Flight) cameras. The 3D point cloud data acquired by 3D sensors is a sparse collection of points, and simply looking at it makes it difficult to determine which 3D point cloud corresponds to which structure (building or facility). Furthermore, unlike regular image data, 3D point cloud data is structured data, so it has a large data volume, making it difficult to share the situation of a site, etc., represented by the 3D point cloud data, in real time.

[0003] Patent Document 1 discloses measuring the distance to each point within a predetermined area including an object using a distance sensor. Patent Document 1 also discloses converting the measurement data of each point into mesh data using a three-dimensional object recognition means and grouping this mesh data into multiple mesh groups. Patent Document 1 also discloses creating a projection plane perpendicular to the normal vector of each mesh group and obtaining projection data by projecting each mesh group onto the corresponding projection plane. Patent Document 1 also discloses extracting contour data from the projection data and comparing the contour data with two-dimensional shape data of the object to recognize the position and orientation of the object. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2011-159042 Summary of the Invention

[0005] When 3D point cloud data is meshed, converted into CAD (Computer Aided Design) data, or polygonized, the structure is represented as an object, the data volume is reduced, and the processing speed for drawing and interaction can be increased. However, meshed objects have lower accuracy than the 3D point cloud itself, so they are not suitable for measuring the details of an object with high precision.

[0006] Therefore, the object of the present disclosure is to provide a real-time communication support system, a real-time communication support method, a mobile terminal, a server, and program The purpose is to provide

[0007] The real-time communication support system of the present disclosure includes: a means for photographing the subject; means for acquiring 3D point cloud data from the photographed object; A means for obtaining a mesh screen of a landscape in which everything other than the target is meshed; means for storing the 3D point cloud data of the object and the mesh view of the scene; and means for simultaneously displaying the 3D point cloud data of the object and the mesh screen of the scenery.

[0008] In addition, the real-time communication support system of the present disclosure includes: A means of photographing the landscape, including the subject, means for determining the object from the photographed scene; means for acquiring 3D point cloud data from the photographed scene; a means for generating a mesh screen of the scenery by meshing the 3D point cloud data other than the determined object; means for storing the 3D point cloud data of the object and the mesh view of the scene; and means for simultaneously displaying the 3D point cloud data of the object and the mesh screen of the scenery.

[0009] In addition, the mobile terminal of the present disclosure includes: The mobile terminal includes a means for photographing an object and a means for simultaneously displaying 3D point cloud data of the object and a mesh screen of a landscape in which everything other than the object has been meshed.

[0010] In addition, the server of the present disclosure The server comprises a means for acquiring 3D point cloud data from a photographed object, a means for acquiring a mesh screen of a landscape in which everything other than the object has been meshed, and a means for saving the 3D point cloud data of the object and the mesh screen of the landscape.

[0011] Further, the real-time communication support method of the present disclosure includes: photographing an object; acquiring 3D point cloud data from the photographed object; A step of obtaining a mesh screen of a landscape in which everything other than the target is meshed; saving the 3D point cloud data of the object and the mesh view of the scene; and simultaneously displaying the 3D point cloud data of the object and the mesh screen of the scenery.

[0012] In addition, the program of the present disclosure This is a program that causes a mobile terminal to execute the steps of photographing an object and simultaneously displaying 3D point cloud data of the object and a mesh screen of the scenery in which everything other than the object has been meshed.

[0013] The present disclosure makes it possible to provide a real-time communication support system that simultaneously displays detailed 3D point cloud data and small-volume mesh data. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 10 is a diagram showing distance measurement in infrastructure inspection etc. according to an embodiment. [Figure 2] FIG. 1 is a diagram illustrating a first meshing technique according to an embodiment. [Figure 3] FIG. 10 is a diagram illustrating a second meshing technique according to an embodiment. [Figure 4] FIG. 1 is a diagram illustrating a configuration of a system according to an embodiment. [Figure 5] FIG. 2 is a diagram illustrating a configuration of a mobile terminal according to an embodiment. [Figure 6] FIG. 2 is a diagram illustrating a configuration of a server (cloud) according to an embodiment. [Figure 7] 10 is a flowchart of a process of the mobile terminal according to the embodiment. [Figure 8] 10 is a flowchart of a process of a server (cloud) according to an embodiment. [Figure 9] FIG. 10 is a diagram illustrating the application of the meshing process according to the embodiment to a simulation. [Figure 10] FIG. 10 is a diagram illustrating the application of the meshing process according to the embodiment to logistics. [Figure 11] FIG. 10 is a diagram showing the application of the meshing process according to the embodiment to animal breeding. [Figure 12] FIG. 10 is a diagram illustrating the application of the meshing process according to the embodiment to remote control of a construction machine. [Figure 13] 1 is a block diagram of a communication support system according to an embodiment; DETAILED DESCRIPTION OF THE INVENTION

[0015] Embodiment Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the invention according to the claims is not limited to the following embodiments. Furthermore, not all of the configurations described in the embodiments are necessarily essential means for solving the problems. For clarity of explanation, the following description and drawings have been omitted and simplified as appropriate. In each drawing, the same elements are given the same reference numerals, and duplicate explanations are omitted as necessary.

[0016] (Outline of this embodiment and meshing method) Fig. 1 is a diagram showing remote measurement in infrastructure inspection etc. according to an embodiment. With reference to Fig. 1, an outline of this embodiment will be described taking as an example the maintenance, operation, management, inspection and monitoring of infrastructure facilities in electric power companies and telecommunications companies.

[0017] As shown in Figure 1, when using a 3D sensor to measure the size of utility poles, the length of power lines, or the distance between utility poles and power lines and buildings, the 3D point cloud data can only be viewed on measuring devices such as tablets held by workers at high altitudes. Therefore, supervisors who are not at the measurement site, such as at a remote business office, cannot view the 3D point cloud data. Because 3D point cloud data has a large data volume and a high processing load, it is difficult to transfer the 3D point cloud data to other workers' devices and display and share it in real time. Furthermore, it is difficult for other workers to understand the situation on site if the display is entirely 3D point cloud data.

[0018] Therefore, if the load is reduced by meshing the non-target area, measurement results and comments on them can be shared in real time, making it easier to understand the situation on site. Figure 2 is a diagram showing a first meshing method according to an embodiment. As shown in Figure 2, a pre-prepared mesh of buildings other than the target area can be acquired, and this mesh can be overlaid on the captured 3D point cloud for display. The pre-prepared building mesh may use external data such as city data from PLATEAU.

[0019] Fig. 3 is a diagram showing a second meshing technique according to an embodiment. As shown in Fig. 3, a 3D point cloud of a landscape including a target is photographed, and the landscape other than the target is meshed, and the mesh and the 3D point cloud are displayed superimposed. This makes it easy to process and apply the results, such as recording them as digital information and adding information on the spot.

[0020] (Configuration of the system according to the embodiment) Fig. 4 is a diagram showing the configuration of a system according to an embodiment. The configuration of the system according to this embodiment will be described with reference to Fig. 4. The system includes a server (cloud) 100 and a plurality of mobile terminals 200.

[0021] The server (cloud) 100 is a computer server or a cloud, etc. The server (cloud) 100 is connected to a plurality of mobile terminals 200 via a wired LAN (Local Area Network), a wireless LAN, an internet line, a mobile phone line, or the like.

[0022] The mobile terminal 200 is a personal computer, a tablet, or a smartphone. The mobile terminal 200 may also be a glasses-type device that realizes VR (Virtual Reality), AR (Augmented Reality), or MR (Mixed Reality). The mobile terminal 200 may also be a 3D display. The mobile terminal 200 may be carried by a user. The mobile terminal 200 may also be a system that exists on-site and exchanges information with a user. For example, the mobile terminal 200 is a system that presents information to a user using a projector and acquires information from the user using a sensor or voice. The mobile terminal 200 acquires image data 300 using a 3D sensor and transmits and receives data to other mobile terminals 200 in real time via the server (cloud) 100. The mobile terminal 200 may only acquire data from the server (cloud) 100, like a supervisor's terminal, without acquiring image data 300 using a 3D sensor.

[0023] (Description of a mobile terminal according to an embodiment) Fig. 5 is a diagram illustrating a configuration of a mobile terminal according to an embodiment. The configuration of the mobile terminal according to the embodiment will be described with reference to Fig. 5. The mobile terminal 200 includes a data collection unit 210, an information display unit 211, an information assignment unit 212, and a data transmission unit 213.

[0024] The data collection unit 210 is a part that has the function of acquiring shooting data 300 acquired by a 3D sensor such as a LiDAR, ToF camera, or stereo sensor, an RGB camera, or other sensor, and data stored in the server (cloud) 100. The 3D sensor may capture images while the photographer is moving or while the 3D sensor is moving. Also, multiple 3D sensors may be used.

[0025] The information display unit 211 is a display device such as a liquid crystal display device or an organic EL (Electro Luminescence) display device. The information display unit 211 is connected to the data collection unit and has a function of displaying information acquired by the data collection unit 210. The information display unit 211 displays 3D point cloud data and meshed mesh screen data. The information display unit 211 may be expressed in a virtual space or may display an avatar of a user for easy understanding, but the expression method is not limited to these. Furthermore, the information display unit 211 may not only display the 3D point cloud data and the meshed mesh screen data, but may also overlay them to make them easier to see. Furthermore, the information display unit 211 may switch between the 3D point cloud data and the meshed mesh screen data and adjust the degree of overlap. Furthermore, the information display unit 211 may select whether to display the 3D point cloud data or the meshed data.

[0026] The information adding unit 212 is connected to the information display unit 211 and has a function of adding information to data displayed on the information display unit 211. The information adding unit 212 may update and edit data acquired by the data collection unit 210. The information adding unit 212 also generates information from other functions of the mobile terminal 200 (such as a camera and a global positioning system (GPS)) and generates original information by the user. Specifically, the information adding unit 212 adds information such as location information such as the latitude and longitude of the location where the image was taken, the direction in which the image was taken, and information identifying an object to be displayed as a 3D point cloud. The information adding unit 212 also adds information such as comments about the object. For example, the information adding unit 212 cooperates with a touch sensor, a touch pen, or the like installed on the information display unit 211.

[0027] The data transmission unit 213 is connected to the information providing unit 212 and has a function of transmitting the information provided by the information providing unit 212 to the server (cloud) 100. As described above, the data transmission unit 213 is connected to a wireless LAN, a wired LAN, an internet line, a mobile phone line, or the like.

[0028] (Description of Server (Cloud) According to the Embodiment) Fig. 6 is a diagram illustrating a configuration of a server (cloud) according to an embodiment. The configuration of the server (cloud) according to the embodiment will be described with reference to Fig. 6. The server (cloud) 100 includes a location information acquisition unit 110, a data collection unit 111, an information management unit 112, and a data transmission unit 115. The information management unit 112 includes a mesh processing unit 113 and an information storage unit 114.

[0029] The position information acquisition unit 110 is a part that has a function of acquiring position information of the terminal user, the subject to be photographed, etc. The data collection unit 111 is a part that is connected to the data transmission unit 213 of the mobile terminal 200 and has a function of collecting data transmitted from the mobile terminal 200. The position information acquisition unit 110 is connected to the data collection unit 111 and acquires position information of the terminal user, the subject to be photographed, etc. from the mobile terminal 200. The data collection unit 111 acquires mesh data from an external system or acquires 3D point cloud data from the mobile terminal 200.

[0030] The information management unit 112 is connected to the data collection unit 111 and receives mesh data or 3D point cloud data. The mesh processing unit 113 of the information management unit 112 has the function of converting mesh data or meshing point clouds other than the target. The mesh processing unit processes mesh data acquired from an external source so that it can be applied to this system. Alternatively, the mesh processing unit 113 meshes the 3D point cloud, converts it into polygons, or converts it into CAD data. The mesh processing unit 113 may perform meshing processing in cooperation with an external system. Here, the external system may be a CAD system, a completed drawing management system, a drawing management system, a GIS (Geographic Information System), etc. The meshing processing does not limit the meshing target to a distinction between target and non-target. Furthermore, the meshing processing may determine whether to mesh and the granularity of the meshing based on processing power, the number of point clouds, density, accuracy, error, etc. The granularity of the meshing refers to the number of polygons and the accuracy of the number of operations. In addition, the meshing process may determine the target to be meshed and the granularity of the meshing based on instructions from a person or in cooperation with external data.

[0031] The information storage unit 114 of the information management unit 112 is a magnetic recording medium (e.g., a flexible disk, a magnetic tape, a hard disk drive), a magneto-optical recording medium (e.g., a magneto-optical disk), or a CD-ROM (Read Only Memory). Alternatively, the information storage unit 114 of the information management unit 112 is a CD-R or a CD-R / W. Alternatively, the information storage unit 114 of the information management unit 112 is a semiconductor memory (e.g., a mask ROM, a programmable ROM (PROM), an erasable PROM (EPROM), a flash ROM, or a random access memory (RAM)). The information storage unit 114 of the information management unit 112 has the function of storing 3D point cloud data of the target and data of scenery other than the target meshed by the mesh processing unit 113. The information storage unit 114 may store a temporal history of the data and perform meshing processing based on the history and changes over time. Alternatively, the information storage unit 114 may store meshed data and data not subject to meshing and use the data to determine whether or not a target is to be meshed. Furthermore, the information storage unit 114 may store the data and use it for object detection (searching and matching of objects, etc.) and tracking of objects. These processes can reduce the weight and simplify the point cloud processing. Furthermore, the mobile terminal 200 may include a mesh processing unit 113. The mobile terminal 200 may transmit 3D point cloud data, a mesh, or both to the server (cloud) 100.

[0032] The data transmission unit 115 is connected to the information management unit 112 and the mobile terminal 200. The data transmission unit 115 has a function of transmitting 3D point cloud data of the target stored in the information management unit 112 and data of scenery other than the target meshed by the mesh processing unit 113 to each mobile terminal 200 in response to a request received from each terminal. The data transmission unit 115 is connected to a wireless LAN, a wired LAN, an internet line, a mobile phone line, etc.

[0033] (Description of Flowchart of Mobile Terminal According to Embodiment) 7 is a flowchart of the process of the mobile terminal according to the embodiment, and the flowchart of the mobile terminal according to the embodiment will be described with reference to FIG.

[0034] First, the mobile terminal 200 captures an image of an object or an object and a landscape using a 3D sensor or the like (step A1). Here, the landscape refers to the portion of the image captured by clipping out the object, specifically, a factory or warehouse, a zoo or ranch, a construction site, or a location where infrastructure facilities are installed. Furthermore, the object refers to luggage, an animal, a portion of a construction site that the user wishes to check or a portion of construction equipment that the user wishes to operate, or infrastructure facilities such as utility poles and power lines. Next, the mobile terminal 200 acquires data such as location information and the location of the image capture using the data collection unit 210 (step A2). Next, the mobile terminal 200 displays the information using the information display unit 211 (step A3). If data stored in the server (cloud) 100 is displayed instead of data acquired by the 3D sensor, step A1 may be omitted. The data collection unit 210 may perform point cloud processing, meshing processing, or preprocessing for point cloud processing and meshing processing. Next, the mobile terminal 200 determines an object from the photographed scenery as needed by the information providing unit 212, and provides information such as providing a comment (step A4). Finally, the mobile terminal 200 transmits the data to the server (cloud) 100 (step A5).

[0035] (Description of Flowchart of Server According to Embodiment) 8 is a flowchart of the processing of the server according to the embodiment, which will be described with reference to FIG.

[0036] First, the server (cloud) 100 acquires location information of the terminal and the shooting data (shooting location, shooting direction, shooting settings and conditions, and additional information) using the location information acquisition unit 110 (step B1). Next, the server (cloud) 100 acquires 3D point cloud data of the photographed object or the object and scenery using the data collection unit 111 (step B2). In this step B2, point cloud data from the sensors that captured the images may be combined, and multiple sensors may be synthesized (registered). Alignment of the point cloud data using location information and external data may be performed in this step B2, or may be performed after meshing.

[0037] Next, the server (cloud) 100 acquires a mesh screen using the mesh processing unit 113, or creates a mesh screen by meshing the 3D point cloud data of the scenery other than the determined target (step B3). Meshing may be performed on the mobile terminal 200 at the site, rather than on the server (cloud) 100. The 3D mesh may be created before capturing an image of the target with a 3D sensor and acquiring 3D point cloud data, or after capturing the 3D point cloud data. The created mesh screen may also be matched with a mesh (polygon, CAD information, model, etc.) stored externally or in the information storage unit 114, and aligned and identified. In this step, this is referred to as "meshing." The point cloud processing and meshing processes may include pre-processing and post-processing, such as noise removal and correction.

[0038] Next, the server (cloud) 100 stores all or all of the required processing data in the information storage unit 114 (step B4). For example, the server (cloud) 100 stores the target 3D point cloud data and a mesh screen of the landscape. By updating all or all of the required data (step B5), real-time information is displayed. The granularity and accuracy of the mesh are adjusted based on human instructions, cooperation with external systems, the real-time nature of internal processing, and resource availability, and real-time information is displayed.

[0039] Next, the server (cloud) 100 adds additional information as needed, such as the information added using the mobile terminal 200 in step A4 (step B6). Finally, the server transmits data such as the 3D point cloud data of the target and a mesh screen of the scenery to each mobile terminal 200 (step B7). The information display unit 211 simultaneously displays the 3D point cloud data of the target and the mesh screen of the scenery sent to each mobile terminal 200.

[0040] According to this embodiment, when acquiring a point cloud using a 3D sensor to measure the distance to a utility pole or power line, it is possible to display an image by meshing everything other than the target and displaying only the target as a point cloud. This makes the operation screen easier to understand, reduces the data volume and processing load, and allows images to be displayed in real time and shared with other devices.

[0041] (Explanation of application of meshing processing of this embodiment to simulation) 9 is a diagram showing the application of the meshing process according to the embodiment to a simulation. With reference to FIG. 9, the application of the meshing process according to the embodiment to a simulation will be described.

[0042] As shown in Figure 9, the point cloud data is clustered (divided into segments) or meshed into a model object, which is then used for simulation. For example, when measuring the distance between power lines or communication lines, if the captured 3D data shows a distance violation, the 3D point cloud or model of the existing power lines can be moved. In this way, it is possible to find a location where the distance violation can be resolved. A function for automatically finding a location where the distance violation can be resolved may also be provided.

[0043] (Description of application of meshing processing of this embodiment to logistics) 10 is a diagram showing application of the meshing process according to the embodiment to physical distribution. With reference to FIG. 10, application of the meshing process according to the embodiment to physical distribution will be described.

[0044] As shown in FIG. 10 , the meshing process according to this embodiment can be applied to logistics and warehouse operations management. For example, the meshing process according to this embodiment can be used in situations such as measuring the size of packages on a conveyor belt in a factory or warehouse or checking for abnormalities. If a 3D sensor were to acquire and measure all of the 3D point cloud data for packages in a factory or warehouse, the data volume would be large, the processing load would be high, and because the packages move, real-time display would be impossible. However, by performing meshing processing on objects other than the packages, the data volume would be reduced, allowing packages in the factory or warehouse to be displayed in real time and the data to be shared. For example, an event camera can be used in combination to display moving objects as a 3D point cloud and stationary objects as a mesh.

[0045] (Description of application of meshing processing according to this embodiment to animal breeding) 11 is a diagram showing the application of the meshing process according to the embodiment to animal breeding. With reference to FIG. 11, the application of the meshing process according to the embodiment to animal breeding will be described.

[0046] As shown in FIG. 11 , the meshing process according to this embodiment can be applied to situations where animals are kept, managed, or monitored. Using a 3D sensor to acquire 3D point cloud data to measure the size of animals at a zoo or farm results in a large data volume, a high processing load, and the animals move, making it impossible to display them in real time. However, by performing meshing on the zoo and farm, which are not the subject of interest, the data volume is reduced, allowing the animals at the zoo or farm to be displayed in real time and the data to be shared. For example, an event camera can be used in combination to display moving objects as a 3-point cloud and stationary objects as a mesh.

[0047] (Description of application of meshing processing according to this embodiment to remote control of construction machinery) 12 is a diagram showing application of the meshing process according to the embodiment to remote control of a construction machine. With reference to FIG. 12, application of the meshing process according to the embodiment to remote control of a construction machine will be described.

[0048] As shown in Fig. 12, the meshing process according to this embodiment can be applied to a situation where construction machinery is remotely operated at a construction site. When operating construction machinery remotely, displaying the entire vast site as a 3D point cloud results in a high processing load. Here, by displaying only the part of the construction site that the user wants to check or the part of the construction machinery that the user wants to operate as a 3D point cloud and displaying the rest as a mesh, the overall picture is easier to grasp, the part that the user wants to operate can be displayed in real time with high accuracy, and the data can be shared.

[0049] Furthermore, some or all of the processes in the server (cloud) 100 and the mobile terminal 200 described above can be realized as a computer program. Such a program can be stored in various types of non-transitory computer-readable media and provided to a computer. Non-transitory computer-readable media include various types of tangible recording media. Examples of non-transitory computer-readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)). The program may also be provided to a computer by various types of temporary computer-readable media. Examples of temporary computer-readable media include electrical signals, optical signals, and electromagnetic waves. The temporary computer-readable media can provide the program to a computer via wired communication paths such as electric wires and optical fibers, or via wireless communication paths.

[0050] 13 is a block diagram of a real-time communication support system according to an embodiment of the present invention, which will be described with reference to FIG. The real-time communication support system 10 according to this embodiment comprises a means 11 for photographing an object, a means 12 for acquiring 3D point cloud data from the photographed object, a means 13 for acquiring a mesh screen of a landscape in which everything other than the object has been meshed, a means 14 for saving the 3D point cloud data of the object and the mesh screen of the landscape, and a means 15 for simultaneously displaying the 3D point cloud data of the object and the mesh screen of the landscape.

[0051] This embodiment can provide a real-time communication support system that simultaneously displays detailed 3D point cloud data and small-volume mesh data.

[0052] Although the embodiments of the present invention have been described above, the present invention includes appropriate modifications that do not impair the objects and advantages thereof, and is not limited to the above-described embodiments.

[0053] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes. (Appendix 1) a means for photographing the subject; means for acquiring 3D point cloud data from the photographed object; A means for obtaining a mesh screen of a landscape in which everything other than the target is meshed; means for storing the 3D point cloud data of the object and the mesh view of the scene; and means for simultaneously displaying the 3D point cloud data of the object and the mesh screen of the scenery. (Appendix 2) a mobile terminal including a means for photographing the object and a means for simultaneously displaying the 3D point cloud data of the object and the mesh screen of the scenery; A real-time communication support system as described in Appendix 1, comprising: a server having: means for acquiring 3D point cloud data from the photographed object; means for acquiring the mesh screen of the scenery in which everything other than the object has been meshed; and means for saving the 3D point cloud data of the object and the mesh screen of the scenery. (Appendix 3) A means of photographing the landscape, including the subject, means for determining the object from the photographed scene; means for acquiring 3D point cloud data from the photographed scene; a means for generating a mesh screen of the scenery by meshing the 3D point cloud data other than the determined object; means for storing the 3D point cloud data of the object and the mesh view of the scene; and means for simultaneously displaying the 3D point cloud data of the object and the mesh screen of the scenery. (Appendix 4) a mobile terminal including: a means for photographing a landscape including the object; a means for determining the object from the photographed landscape; and a means for simultaneously displaying the 3D point cloud data of the object and the mesh screen of the landscape; A real-time communication support system as described in Appendix 3, comprising a server having: means for acquiring 3D point cloud data from the photographed scenery; means for meshing the 3D point cloud data other than the determined object to create the mesh screen of the scenery; and means for saving the 3D point cloud data of the object and the mesh screen of the scenery. (Appendix 5) A real-time communication support system according to any one of appendices 1 to 4, comprising means for simulating the movement of the 3D point cloud data of the object relative to the mesh screen of the scenery. (Appendix 6) The scene is a factory or warehouse and the object is a package, or the scene is a zoo or farm and the objects are animals, or The scene is a construction site, and the object is a part of the construction site that the user wants to check or a part of the construction site that the user wants to operate construction equipment on, or 6. A real-time communication support system according to any one of claims 1 to 5, wherein the scenery is a location where infrastructure equipment is installed, and the target is the infrastructure equipment. (Appendix 7) A mobile terminal comprising: a means for photographing an object; and a means for simultaneously displaying 3D point cloud data of the object and a mesh screen of a landscape in which everything other than the object has been meshed. (Appendix 8) A server comprising: a means for acquiring 3D point cloud data from a photographed object; a means for acquiring a mesh screen of a landscape in which everything other than the object has been meshed; and a means for saving the 3D point cloud data of the object and the mesh screen of the landscape. (Appendix 9) photographing an object; acquiring 3D point cloud data from the photographed object; A step of obtaining a mesh screen of a landscape in which everything other than the target is meshed; saving the 3D point cloud data of the object and the mesh view of the scene; and simultaneously displaying the 3D point cloud data of the object and the mesh screen of the scenery. (Appendix 10) A non-transitory computer-readable medium that records a program that causes a mobile terminal to execute the steps of photographing an object and simultaneously displaying 3D point cloud data of the object and a mesh screen of a landscape in which everything other than the object has been meshed. (Appendix 11) A non-transitory computer-readable medium recording a program to be executed by a server, the steps of acquiring 3D point cloud data from a photographed object, acquiring a mesh screen of a landscape in which everything other than the object has been meshed, and saving the 3D point cloud data of the object and the mesh screen of the landscape. (Appendix 12) A mobile terminal comprising: a means for photographing a landscape including an object; a means for determining the object from the photographed landscape; and a means for simultaneously displaying 3D point cloud data of the object and a mesh screen of the landscape other than the object. (Appendix 13) A server comprising: a means for acquiring 3D point cloud data from a landscape including a photographed object; a means for meshing the 3D point cloud data of the landscape other than the object to create a mesh screen of the landscape; and a means for saving the 3D point cloud data of the object and the mesh screen of the landscape. (Appendix 14) Photographing a landscape including a subject; determining the object from the captured scene; acquiring 3D point cloud data from the captured scene; a step of meshing the 3D point cloud data other than the determined target to create a mesh screen of the scenery; saving the 3D point cloud data of the object and the mesh view of the scene; and simultaneously displaying the 3D point cloud data of the object and the mesh screen of the scenery. (Appendix 15) A real-time communication support method as described in Appendix 14, which maintains a temporal history of the step of saving the 3D point cloud data of the object and the mesh screen of the scenery, and performs a step of meshing the 3D point cloud data other than the determined object based on the change in the history over time to create a mesh screen of the scenery. (Appendix 16) A non-transitory computer-readable medium that records a program that causes a mobile terminal to execute the following steps: photographing a landscape including an object; determining the object from the photographed landscape; and simultaneously displaying 3D point cloud data of the object and a mesh screen of the landscape other than the object. (Appendix 17) A non-transitory computer-readable medium that records a program that causes a server to execute the following steps: acquiring 3D point cloud data from a landscape including a photographed object; meshing the 3D point cloud data of the landscape other than the object to create a mesh screen of the landscape; and saving the 3D point cloud data of the object and the mesh screen of the landscape.

[0054] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the invention.

[0055] This application claims priority based on Japanese Patent Application No. 2022-051437, filed on March 28, 2022, the disclosure of which is incorporated herein in its entirety. [Explanation of symbols]

[0056] 10 Real-time communication support system 11 Means of photographing the subject 12 Means of acquiring 3D point cloud data 13 How to get a mesh screen 14. Means of saving 3D point cloud data and mesh images 15. A method for simultaneously displaying 3D point cloud data and mesh screens 100 servers (cloud) 110 Location information acquisition unit 111 Data Collection Department 112 Information Management Department 113 Mesh processing section 114 Information Holdings Department 115 Data transmission unit 200 mobile devices 210 Data Collection Department 211 Information display section 212 Information Assignment Department 213 Data Transmission Unit

Claims

1. a means for photographing the subject; means for acquiring 3D point cloud data from the photographed object; A means for obtaining a mesh screen of a landscape in which everything other than the target is meshed; means for storing the 3D point cloud data of the object and the mesh view of the scene; means for simultaneously displaying the 3D point cloud data of the object and the mesh image of the scenery; and means for simulating the movement of the 3D point cloud data of the object relative to the mesh screen of the scenery.

2. a mobile terminal including a means for photographing the object and a means for simultaneously displaying the 3D point cloud data of the object and the mesh screen of the scenery; The real-time communication support system of claim 1, comprising a server having: means for acquiring 3D point cloud data from the photographed object; means for acquiring a mesh screen of the scenery in which everything other than the object has been meshed; and means for saving the 3D point cloud data of the object and the mesh screen of the scenery.

3. A means of photographing the landscape, including the subject, means for determining the object from the photographed scene; means for acquiring 3D point cloud data from the photographed scene; a means for generating a mesh screen of the scenery by meshing the 3D point cloud data other than the determined object; means for storing the 3D point cloud data of the object and the mesh view of the scene; means for simultaneously displaying the 3D point cloud data of the object and the mesh image of the scenery; and means for simulating the movement of the 3D point cloud data of the object relative to the mesh screen of the scenery.

4. a mobile terminal including: a means for photographing a landscape including the object; a means for determining the object from the photographed landscape; and a means for simultaneously displaying the 3D point cloud data of the object and the mesh screen of the landscape; The real-time communication support system of claim 3, comprising a server having: means for acquiring 3D point cloud data from the photographed scenery; means for meshing the 3D point cloud data other than the determined object to create the mesh screen of the scenery; and means for saving the 3D point cloud data of the object and the mesh screen of the scenery.

5. The scene is a factory or warehouse and the object is a package, or the scene is a zoo or farm and the objects are animals, or The scene is a construction site, and the object is a part of the construction site that the user wants to check or a part of the construction site that the user wants to operate construction equipment on, or The real-time communication support system according to claim 1 , wherein the scenery is a location where infrastructure facilities are installed, and the object is the infrastructure facilities.

6. a means for photographing an object, and a means for simultaneously displaying 3D point cloud data of the object and a mesh screen of a landscape in which everything other than the object is meshed; and means for simulating the movement of the 3D point cloud data of the object relative to the mesh screen of the scene.

7. A means for acquiring 3D point cloud data from a photographed object, a means for acquiring a mesh screen of a landscape in which everything other than the object is meshed, and a means for saving the 3D point cloud data of the object and the mesh screen of the landscape; and means for simulating the movement of the 3D point cloud data of the object relative to the mesh view of the scene.

8. photographing an object; acquiring 3D point cloud data from the photographed object; A step of obtaining a mesh screen of a landscape in which everything other than the target is meshed; saving the 3D point cloud data of the object and the mesh view of the scene; simultaneously displaying the 3D point cloud data of the object and the mesh view of the scene; and simulating the movement of the 3D point cloud data of the object relative to the mesh screen of the scenery.

9. a step of photographing an object; and a step of simultaneously displaying 3D point cloud data of the object and a mesh screen of a landscape other than the object, the mesh screen being a mesh of the landscape; and a step of simulating the movement of the 3D point cloud data of the object relative to the mesh screen of the scenery.

Images