Remote presence support system

Abstract

To provide a remote presence support system which contributes to achievement of high quality remote presence.SOLUTION: A remote presence support system 1 which controls communication among a plurality of information apparatuses A, to execute a web conference while viewing a 360-degree video P acquired by using an imaging apparatus 9 on a site intended for remote presence and supports the remote presence through the web conference includes: web conference screen display means for allowing the output parts of the plurality of information apparatuses A to display a web conference screen S including the 360-degree video P; storage means for storing the 360-degree video; and instruction information sharing means for allowing the output parts of the information apparatuses A relating to an instruction information sharing instruction to output instruction information, when the instruction information obtained by adding additional information 10 to the 360-degree video P stored in the storage means is created by using one of the plurality of information apparatuses A and after that, the instruction information sharing instruction for sharing the instruction information is received from the information apparatus A.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a remote presence support system suitable for use in remote presence support, for example. [Background technology]

[0002] In the construction industry, witness inspections (stage confirmation inspections) are carried out for each type and process of construction. Participants in witness inspections include the building owner, contractor, designer, supervisor, and local government officials. At the site where the witness inspection is carried out, the appearance, structure, materials, construction methods, safety measures, etc. of the building are checked, and any deficiencies or problems are pointed out.

[0003] Meanwhile, against the backdrop of the recent worsening shortage of workers (engineers) in the construction industry, tasks that require participants to gather on-site, such as witness inspections, are seen as wasteful travel by participants to and from the site, and there is a strong demand for efficiency improvements.

[0004] In light of this situation, remote on-site inspections using web conferencing systems (e.g., Patent Document 1) have begun to be implemented in recent years instead of witness inspections. In remote on-site inspections, participants at the site (e.g., site supervisors) take pictures of the state of the construction site using, for example, a small camera mounted on a smartphone or PC, and participants at remote locations (e.g., clients and supervisors) view the images to check whether the construction is being carried out appropriately. In recent years, the use of 360-degree cameras to capture and record the entire surroundings has also become widespread. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent No. 6964371 Summary of the Invention [Problem to be solved by the invention]

[0006] However, remote on-site inspections have the following problems compared to traditional on-site witness inspections. Experienced personnel (workers, supervisors, etc.) with plenty of experience in on-site witness inspections can find points that need to be checked even in remote on-site inspections based on past experience, but unskilled personnel may miss points that need to be checked in witness inspections or may not notice places that are often pointed out, and may not be able to respond appropriately. In particular, in the case of small cameras with a typical angle of view used by on-site participants, there are significant blind spots in the range that can be checked, and participants in remote locations may miss areas that need to be inspected. Even if a blind spot is noticed, participants in remote locations must be instructed to change the direction of the camera in order to check the area in the blind spot, which can take time and effort to give instructions from the remote participants to the on-site participants, or the instructions may not be conveyed correctly, increasing the possibility of overlooking important parts. In addition, participants in remote locations may not notice things that would be noticed if they were on-site (for example, temperature or size of objects), and may not be able to point them out adequately.

[0007] The present invention has been made in consideration of the above-mentioned circumstances, and an object of the present invention is to provide a remote presence support system that contributes to realizing a high-quality remote presence. [Means for solving the problem]

[0008] In order to achieve the above object, the remote presence support system according to the present invention comprises: A remote presence support system that controls communication between a plurality of information devices to enable a web conference to be held while viewing a 360-degree image captured by an imaging device at a site to be remotely supported, thereby enabling remote support through a web conference, a web conference screen display means for displaying a web conference screen including the 360-degree video on an output unit of the plurality of information devices; A storage means for storing the 360-degree video; After instruction information is created by adding additional information to the 360-degree image stored in the storage means using one of the plurality of information devices, an instruction information sharing instruction for sharing the instruction information is received from the information device, and an instruction information sharing means causes an output section of the information device related to the instruction information sharing instruction to output the instruction information (Claim 1).

[0009] In the remote presence support system, the storage means stores the 360-degree video and the additional information constituting the instruction information in a mutually linked state, The information processing device may further include a past additional information adding means for adding, upon receiving from one of the plurality of information devices, an instruction to add additional information linked to past 360-degree video captured at the same position to any 360-degree video used for remote presence, the additional information being stored in the storage means to the any 360-degree video (Claim 2).

[0010] The remote presence support system may also have a past video sharing means for including, when receiving a past video sharing instruction from one of the plurality of information devices for sharing a specified past 360-degree video stored in the storage means, the specified past 360-degree video on a web conference screen of the information device related to the past video sharing instruction (Claim 3).

[0011] In the remote presence system, at least one of the plurality of information devices may be a head mounted display (claim 4).

[0012] The remote presence support system may have a thermography overlay display means for overlaying a thermography obtained by measuring the surrounding temperature using a thermography camera placed at the shooting point of the imaging device at the site on a 360-degree image (Claim 5).

[0013] The remote presence support system may have a three-dimensional data synthesis means for overlaying three-dimensional data obtained by three-dimensionally scanning the surroundings using a LiDAR sensor placed at the shooting point of the imaging device at the site onto a 360-degree image (Claim 6). Effect of the Invention

[0014] The present invention provides a remote presence support system that contributes to realizing a high-quality remote presence.

[0015] That is, in the remote presence support system according to the invention of each claim of the present application, even if a remote presence participant is in a remote location far from the site, the remote presence participant can freely check the site in all directions through the 360-degree video, which makes it difficult for important parts to be overlooked, etc. Also, each participant can add additional information to the 360-degree video and output it to the output unit of the information device of the other participants, which makes it easy to convey instructions (annotation information), etc.

[0016] In the remote presence support system of the invention according to claim 2, for example, when a remote presence of a certain site is performed, if the site has been remotely observed before, the additional information used in the past remote presence of the site can be added to any 360-degree video used in the current remote presence and referenced, so that the instructions given in the past remote presence can be easily and effectively used in the current remote presence. Specifically, by referring to the past instruction information and displaying the parts where instructions were given in the past synthesized on the current 360-degree video, participants can explicitly check the parts where instructions were given in the past and prevent overlooking the parts where instructions were given. In addition, non-experts can learn the parts where instructions were given in past remote presences, and the learning effect enables high-quality remote presence.

[0017] Non-skilled personnel (workers, supervisors, etc.) may miss points that should be checked during witness inspections, or may not notice places that are often pointed out, and may not be able to respond appropriately. However, the remote on-site support system of the invention of claim 3 stores 360-degree video and makes it possible to check past on-site information, thereby increasing the opportunities to check and analyze inspection items and points pointed out in past inspections, and providing more thorough support to non-skilled personnel.

[0018] In the remote presence support system of the invention according to claim 4, participants can wear a head-mounted display and view 360-degree video, which allows them to check the surrounding situation of the site while placing their viewpoint at the position of the on-site imaging device, and thus makes it easy to understand the relative positions of structures at the site. At this time, it may be possible to electronically write (add) additional information such as instruction lines and characters to the 360-degree video by using a hand controller corresponding to the head-mounted display or gestures by hand tracking.

[0019] Temperature is important when pouring concrete, and when the temperature at the work site gets too high in summer, the risk of health damage to workers increases. With the remote on-site support system of the invention of claim 5, it is possible to use thermography to check the on-site temperature and the body temperature of workers, even from a remote location, and then issue appropriate instructions.

[0020] In the remote presence support system of the invention of claim 6, a LiDAR sensor is used to measure distance by reflecting light, and 3DCG is achieved by acquiring the position, shape, and color of an object, while real-world scale information can be imparted to the 360-degree image. This makes it possible to measure distance by specifying two points in the 360-degree image, or to calculate area, volume, etc. by connecting multiple points. [Brief description of the drawings]

[0021] [Figure 1]FIG. 1A is a conceptual diagram for explaining a remote presence support system of the present invention, and FIG. 1B is a block diagram showing a basic configuration of an information device constituting the remote presence support system. [Diagram 2] FIG. 2 is an overall explanatory diagram of a usage state of the remote presence support system. [Diagram 3] FIG. 2 is a diagram illustrating a main part of the remote presence support system in use; [Figure 4] 6(A) to 6(F) are diagrams illustrating a past additional information adding means of the remote presence support system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] An embodiment of the present invention will be described below.

[0023] FIG. 1(A) is a conceptual diagram for explaining a remote presence support system 1 of the present invention. The information devices A (A1 to A3) constituting this system 1 are computer-based processing devices. The computer may be, for example, any one or a combination of two or more of a portable information terminal (a wearable device such as a smartphone, a tablet, a notebook PC, a head-mounted display, smart glasses, or a smart watch) or a stationary information terminal (a desktop PC, a server, or the like). In addition, the computer can transmit and receive information via a network 2 such as the Internet (intranet) wirelessly or via wire, and when multiple computers are used, functions and roles may be shared, such as by giving some functions to some computers. Furthermore, the computer may be mounted on, for example, a drone.

[0024] In the example of FIG. 1(A), the information devices A constituting the remote presence support system 1 are an information device (distribution terminal) A1 used at the site where the remote presence is to be performed, a total of four (one example of a plurality of) information devices (meeting terminals) A2 (A2a to A2d) used at a remote location away from the site, and a server (cloud server) A3 used and managed at a remote location away from the site (see also FIG. 2). These are only examples, and for example, the number of information devices A2 used at a remote location away from the site may be two or less, or four or more, instead of three. Also, if the function of the server A3 can be provided to an information device A other than the server A3, the server A3 may not be used.

[0025] FIG. 1B is a block diagram showing the basic configuration of information device A. That is, information device A includes a CPU (processor) 3 (controlling the calculation section and the control section) that performs various calculations and control, a communication section 4 that transmits and receives various information to and from other information devices A via a network 2, an input section 5 that is composed of a keyboard, an operation button, a camera, a microphone, etc. and inputs various information, an output section 6 that is composed of a display, a speaker, etc. and outputs various information, and a storage section 7 that stores information (OS, programs, etc.) necessary for calculation and control by CPU 3 and the results of calculation processing, and these elements are connected by a bus 8 or the like and are configured to be able to transmit and receive information. For example, when predetermined information is input from the input section 5, CPU 3 reads out a program corresponding to the input information from the storage section 7 and executes it, and with this execution, information is transmitted and received between each section 4 to 7, and various processing is performed. The various processing is performed in this way by each means such as a web conference screen display means, a storage means, an instruction information sharing means, a past additional information adding means, a past video sharing means, a thermography superimposition display means, and a three-dimensional data synthesis means, which will be described later. Each information device A does not necessarily have to have all of the elements 3 to 8, and it is also possible to use an information device A having only some of the elements 3 to 8. Each information device A may also have a device or the like that serves multiple elements, such as a touch panel that serves both as the input unit 5 and the output unit 6.

[0026] The remote presence support system 1 enables a web conference using the information device A as described above, and the configuration of the web conference itself can use the configuration of a known web conference system (for example, a system that adopts a client-server connection method). Specifically, for example, the video and audio of the user of the information device A1 shown in FIG. 1(A) are input by the input unit 5 (camera, microphone) of the information device A1, and then output to the output unit 6 (display, speaker) of the other information devices A2 (A2a to A2d) via the server A3 through the network 2. Among them, the video of the user is included in the web conference screen S (see FIG. 2) displayed on the display of the output unit 6. The video and audio of the user of the information device A2 are also output to the output units 6 of the other information devices A1 and A2. For example, such a function may be realized by installing a predetermined application program in some or all of the information devices A1 to A2 other than the server A3, or may be realized by using a browser function.

[0027] The remote presence support system 1 of this example controls communication between multiple information devices A (A1 to A3) to enable a web conference to be held while viewing a 360-degree video P (see FIG. 3) captured by a 360-degree camera (an example of an imaging device) 9 at a site to be remotely attended, and has a web conference screen display means, a storage means, and an instruction information sharing means. These means exist in the information devices A1 to A3. Roughly speaking, the web conference screen display means is a means for displaying a web conference screen S (see FIG. 2) including (a part of) the 360-degree video P on a display unit (display) which is an output unit 6 of the multiple information devices A (A1, A2 in this example). The storage means is a means for storing the 360-degree video P. The instruction information sharing means is a means for outputting the instruction information to an output section of the information device A (in this example, A1, A2) related to the instruction information sharing instruction, upon receiving an instruction information sharing instruction for sharing the instruction information from one of a plurality of information devices A (in this example, A1, A2) for creating instruction information by adding additional information 10 to a 360-degree image P (the instruction information may be created, for example, using an application program installed on the information device A, or using software provided by a cloud service).

[0028] The web conference screen display means displays the 360-degree video P, which was captured on-site using the 360-degree camera 9 and sent to the information device A1, on the display unit of the information device A1 (by including it in the web conference screen S), and also sends it to other information devices A2 via the server A3 and displays it on the display unit of each information device A2 (by including it in the web conference screen S). The 360-degree video P is distributed, for example, by a streaming method (live type). The web conference screen display means also includes programs and the like that enable such display, distribution, and the like.

[0029] If the 360-degree camera 9 is equipped with a microphone, the audio input to the microphone of the 360-degree camera 9 may be sent to another information device A2 via the information device A1 and server A3. This audio information may also be stored in the storage means (for example, the storage unit 7 of the server A3).

[0030] In addition, the web conference screen display means displays the 360-degree video P that has been captured using a 360-degree camera 9 at the site and sent to the information device A1 on the display unit of the information device A1, and also sends it to other information devices A2 via the server A3 and displays it on the display units of each information device A2. The 360-degree video P is not limited to being distributed by a streaming method (live type), for example. For example, before a web conference is held, the 360-degree video P captured using the 360-degree camera 9 at the site may be sent to the server A3 via the information device A1 and stored (uploaded) on the server A3. When the web conference is held, the user operates one of the information devices A (for example, the four information devices A2a to A2d shown in FIG. 1A) used to participate in the web conference to access the server A3 and selects any 360-degree video P. The selected 360-degree video P is displayed (included in the web conference screen S) on the display unit of any (or all) of the information devices A used to participate in the web conference (see FIG. 3).

[0031] Storage by the storage means may be performed, for example, by a storage instruction from information device A1 or A2 on whose display the 360-degree video P to be stored can be viewed, or may be performed automatically, and the storage destination (i.e., the storage means) may include at least server A3 (its memory unit 7). If the video is to be used immediately on information device A1 or A2 other than server A3, it may be configured so that the video is also stored on that device. However, it is of course also possible for the storage means to be another device that can send and receive information to and from information device A via network 2.

[0032] The instruction information sharing means executes, for example, the following steps (S1) to (S3) (including programs and the like for that purpose). (S1) In information device A1, video (for a certain period of time) or still images are stored for the real-time 360-degree video P sent from the 360-degree camera 9. (S2) Using the input unit 5, output unit 6, etc., in information device A1, instruction information (composite data) is created by adding additional information 10 such as text to the video or still images stored in (S1) above. (S3) The instruction information is sent from information device A1 via server A3 to any specified information device A2 (or all information devices participating in the web conference), and is displayed on the display unit of the destination information device A2 (included in the web conference screen S).

[0033] In the storage means in this example (for example, memory unit 7 of server A3), the 360-degree video P and additional information 10 that constitute the instruction information are stored in a linked state (note that, if they are linked, the 360-degree video P and additional information 10 may be stored in separate locations (storage means)). In this case, if it is possible to combine the linked 360-degree video P and additional information 10 and output them as instruction information, then storage of the instruction information itself can be omitted, and if such output is not possible, the instruction information itself may be stored.

[0034] Examples of the additional information 10 include handwritten lines 10a (see FIG. 2) and characters inputted with a tablet pen or the like, images and videos such as an electronic small blackboard 10b (see FIG. 2) stored or created in advance in an information device A1 or the like, 2D / 3DCG, BIM / CAD data, electronic files created with a spreadsheet software or word processing software, text and figures that can be inputted with a keyboard, etc. When creating instruction information, the additional information 10 to be added to the 360-degree video P is not limited to the above-mentioned visual information, and may also be audio data (auditory information) or the like (for example, when the instruction information is opened, a specific audio may be played, or a conversation recorded when the remote presence support system was used in the past may be played). In addition, if the additional information 10 is audio data, video, electronic files, etc., then as shown in Figure 2, a mark M such as an icon or symbol corresponding to the information (marks M1 to M3 correspond to audio data, video, and electronic files, respectively) can be placed at an appropriate position on the 360-degree video P (although Figure 2 shows mark M outside the 360-degree video P, in reality it is placed within the 360-degree video P). For example, operating mark M1 (e.g., clicking) will play a specified audio, operating mark M2 will play a specified video, and operating mark M3 will open a specified electronic file.

[0035] In steps (S1) to (S3), the instruction information is created by the information device A1, but the same applies when the instruction information is created by another information device A2. In either case, when adding visual information as additional information to a moving image (or a still image), the additional information (visual information) may be added directly or by placing a transparent layer.

[0036] In the above steps (S1) to (S3), the information device A, which is the destination of the instruction information, only receives the completed instruction information, but this is not limited thereto. If the additional information constituting the instruction information is only visual information, the instruction information may be created in the destination information device A. That is, the instruction information sharing means may, for example, execute the following steps (S'1) to (S'3) (including a program therefor). (S'1) On the display unit of the information device A1, which displays the real-time 360-degree video P sent from the 360-degree camera 9, an operation is performed to input and superimpose additional information 10 on a transparent layer set on the 360-degree video P. (S'2) The coordinates in the 360-degree video P of the additional information 10 input in the above (S'1) are obtained. (S'3) The additional information 10 and coordinates are sent from information device A1 via server A3 to any specified information device A2 (or to all information devices A2 participating in the web conference), and the destination information device A2 displays the received additional information 10 superimposed on the specified coordinate position of the layer placed on the 360-degree image P displayed on the display unit.

[0037] In steps (S'1) to (S'3), the instruction information is created by the information device A1, but the same applies when the instruction information is created by another information device A2.

[0038] Here, as shown on the left side of Fig. 3, the 360-degree video P acquired by the 360-degree camera 9 has a horizontal angle of view of 360 degrees and a vertical angle of view of 180 degrees, and although the entire video may be displayed on the display unit (web conference screen S) of the information device A, in this example, only a portion of the video (for example, a horizontal angle of view of about 90 degrees and a vertical angle of view of about 90 degrees) is displayed in consideration of visibility (see the web conference screen S in the upper right of Fig. 3), and for example, if the display unit of the information device A is a touch panel (touch pad), the range of the 360-degree video P displayed on the display unit can be changed by a flick operation or a swipe operation as if the orientation of the 360-degree camera 9 had been changed (the user can freely move in the direction they want to view). The lower right of Fig. 2 shows an example in which the range of the 360-degree video P displayed on the information device A2d is different from the range displayed on the other information devices A. Also, in this example, the range (horizontal and vertical angles of view) of the 360-degree image P displayed on the display unit (web conference screen S) of information device A can be changed (enlarged and reduced display possible) by zooming in and out, and this zooming in and out operation can be performed by operations using the zoom buttons displayed on the display unit (web conference screen S) (click operations, etc.) or operations using the input unit of information device A (for example, mouse wheel operations or pinch-in and pinch-out operations on a touch panel).

[0039] In this example, the range of the 360-degree video P displayed on the display unit can be different for each information device A, and each information device A may display in real time an indication of which range of the 360-degree video P is displayed on the display unit of the other information device A participating in the web conference (for example, a colored frame assigned to each information device A or a mark indicating the viewpoint, etc.). Also, the range of the 360-degree video P displayed on the display unit may be synchronized between information devices A participating in the web conference.

[0040] The remote presence support system 1 of this embodiment described above contributes to realizing a high-quality remote presence. That is, even if a participant of the remote presence is in a remote location far from the site, it is possible to easily check all directions of the site through the 360-degree video P, which makes it difficult for important parts to be overlooked. Also, each participant can add additional information to the 360-degree video (electronically draw a line, write a character, or place a mark M on a part that they want other participants to pay attention to) and display it on the output unit of the information device A (for example, display it on the web conference screen S), which makes it easy to convey instructions (annotation information), etc.

[0041] In witness inspections traditionally conducted in the construction industry, in order to leave a record, a blackboard on which the construction status and the like (for example, the name of the construction site, the location where the photo was taken, the date of the photo, the company name, the type of work, etc.) is written is photographed together with the subject of the photo. In recent years, photographs are taken using electronic small blackboards created using electronic data, and these photographs are left as records. Incidentally, when taking photographs of electronic small blackboards with blackboards to record during witness inspections, it is necessary to take detailed photographs at each angle of view, but there is a problem in that it is difficult to know later which part was photographed when photographs are taken at each angle of view, and it is necessary to make a note of the photographing position on a map or drawing.

[0042] In this respect, the remote presence support system 1 of this example can solve the above problem since a wide range (360-degree range) including the electronic small whiteboard 10b can be photographed and stored as additional information 10 at once. The electronic small whiteboard 10b is generated by inputting necessary information using one of the information devices A, and it is considered that this is added (combined) to the 360-degree video P acquired by the 360-degree camera 9 during shooting or during remote presence in this example. In order to add (combine) the electronic small whiteboard 10b during shooting, the electronic small whiteboard 10b is placed at an arbitrary position on the display unit of the information device A1 while the 360-degree video P is displayed on the display unit of the information device A1, and when, for example, a shooting button displayed on the display unit of the information device A1 is pressed (a shooting instruction is issued to the 360-degree camera 9), shooting by the 360-degree camera 9 is started, and the 360-degree video including the electronic small whiteboard 10b is stored in the information device A1, the camera 9, and the server A3. When photographing the electronic blackboard 10b during remote attendance (after photographing with the 360-degree camera 9), the electronic blackboard 10b can be placed at any position on the display unit of each information device A while the 360-degree image P captured in advance by the 360-degree camera 9 and received via the server A3 is displayed on the display unit of that information device A, and when, for example, a save button displayed on the display unit of that information device A is pressed (a save operation is executed), the electronic blackboard 10b can be composited into the 360-degree image P at 360 degrees or any arbitrary angle of view, and the composite image can be saved on the information device A (and / or server A3) that pressed the button.

[0043] The information device A2b shown in FIG. 2 is a head-mounted display (including a VR terminal such as VR goggles), and at least one of the multiple information devices A may be a head-mounted display. In this case, the participants can wear the head-mounted display and view the 360-degree video P, which allows them to check the surrounding situation of the site while placing their viewpoint at the position of the 360-degree camera 9 at the site, and thus makes it easy to understand the positional relationship of structures at the site. At this time, it may be possible to electronically write (combine) lines, characters, etc. for instructions on the 360-degree video using a hand controller or hand gestures corresponding to the head-mounted display. In addition, in the case of a head-mounted display, the image does not move, but the wearer moves his or her neck and turns his or her gaze in the direction he or she wants to see, and at this time, a mark or a gaze frame that indicates the direction other participants are looking may be displayed in the video.

[0044] It is to be noted that the present invention is not limited to the above-described embodiment, and can be modified in various ways without departing from the spirit and scope of the present invention. For example, the following modifications can be mentioned.

[0045] The remote presence support system 1 may have a means for superimposing a depth map image taken by a LiDAR sensor and a 360-degree image P, a means for superimposing a heat map image taken by a thermographic camera and a 360-degree image P, a means for superimposing the 360-degree image P with point cloud data, 3DCG data, 3DCAD, etc., and may also use two or more cameras at the site to obtain stereo images that allow for three-dimensional viewing with images spaced apart on the left and right, and use these stereo images as the 360-degree image P.

[0046] That is, a LiDAR sensor (not shown) may be placed at (near) the shooting point of the 360-degree camera 9 on-site, and the remote presence support system 1 may have a three-dimensional data synthesis means (this means is present in the information devices A1 to A3) that scans the surroundings three-dimensionally with the LiDAR sensor and superimposes the three-dimensional data obtained on the 360-degree video P. In this case, the LiDAR sensor measures distance by reflecting light, and obtains the position, shape, and color of an object to create a 3DCG image, while giving the 360-degree video P real-world scale information, so that it is possible to measure distance by specifying two points in the 360-degree video P, or calculate area, volume, etc. by connecting multiple points.

[0047] In this case, the remote presence support system 1 may also have a 3DCG data overlay display means (which means exists in the information devices A1 to A3) that overlays 3DCG data having scale information (e.g., data created in advance using 3D CAD or the like) on the 360-degree video P. By configuring in this way, the 3DCG having scale information can be overlaid on the 360-degree video P at the correct scale, which contributes to improving convenience.

[0048] Furthermore, a thermographic camera (not shown) may be placed at the site at (near) the shooting point of the 360-degree camera 9, and the remote on-site support system 1 may have a thermographic superimposition display means (this means exists in the information devices A1 to A3) that measures the surrounding temperature with this thermographic camera and superimposes the thermogram obtained on the 360-degree video P. Temperature is important when pouring concrete, and when the site temperature becomes high in summer, there is an increased risk of health damage to workers. However, the remote on-site support system 1 having a thermographic superimposition display means makes it possible to issue appropriate instructions after checking the site temperature and the body temperature of workers through thermography, even from a remote location.

[0049] Similarly, sensors that measure humidity, odor, gas, illuminance, vibration, pressure, etc. may be placed at (or near) the shooting point of the 360-degree camera 9, and a superimposed display means may be provided that superimposes and displays the measurement results on the 360-degree image P.

[0050] When the remote on-site support system 1 receives a past video sharing instruction from one of the information devices A (e.g., A1 or A2) for sharing a predetermined past 360-degree video P stored in the storage means from the information devices A, the remote on-site support system 1 may have a past video sharing means (this means exists in the information devices A1 to A3) for including (displaying) the predetermined past 360-degree video P in the web conference screen S of the other information device A (e.g., A1 or A2) related to the past video sharing instruction. Non-skilled personnel (workers, supervisors, etc.) may miss points that should be checked during witness inspections, or may not notice places that are often pointed out and may not be able to respond appropriately. However, the remote on-site support system 1 having the past video sharing means can check past on-site information using the 360-degree video P stored and accumulated in the storage means, so that it is possible to increase opportunities to check and analyze inspection items and pointed out points in past inspections, and to provide thorough support to non-skilled personnel. The 360-degree video P shared by the past video sharing instruction may be instruction information to which additional information has been added.

[0051] In this case, the remote presence support system 1 may have a means for transparently superimposing the current image and the past image. In other words, 360-degree images P of the same direction and angle range are superimposed, and one or both are displayed semi-transparently. This allows the difference between the past and the present to be compared.

[0052] As described above, the storage means stores the 360-degree image P and the additional information 10 that constitute the instruction information in a state in which they are linked to each other. Utilizing this, the remote presence support system 1 may have a past additional information adding means for adding the additional information 10 stored in the storage means to any 360-degree image P used for remote presence, when the remote presence support system 1 receives a past additional information adding instruction from one of a plurality of information devices A to add additional information 10 linked to a past 360-degree image P captured at the same position to the any 360-degree image P.

[0053] Specifically, for example, on one occasion, a remote presence is performed using the remote presence support system 1, and a 360-degree image Px shown in Figure 4(A) and instruction information Ix with additional information 10x added to this image Px are created (see Figure 4(B)); on a different occasion, a remote presence is performed using the remote presence support system 1 at the same site (position) and a 360-degree image Py shown in Figure 4(C) and instruction information Iy with additional information 10y added to this image Py are created (see Figure 4(D)); and after these two occasions, a remote presence is performed using the remote presence support system 1 at the same site (position) and a 360-degree image Pz shown in Figure 4(E) is obtained, and the additional information 10x, 10y are added to this image Pz to create instruction information Iz.

[0054] In this way, for example, when a remote visit to a certain site is performed, if the site has been remotely visited before, the additional information 10 used in the past remote visit to the site can be added to any 360-degree video P used in the current remote visit and referenced, so that the instructions given in the past remote visit can be easily and effectively used in the current remote visit. Specifically, by referring to the past instruction information and displaying the parts where instructions were given in the past synthesized on the current 360-degree video P, participants can explicitly check the parts where instructions were given in the past and prevent overlooking the parts where instructions were given. In addition, non-experts can learn the parts where instructions were given in past remote visits, and the learning effect enables high-quality remote visits.

[0055] In this case, it is possible to keep the installation position of the 360-degree camera 9 used in the past and present constant by marking it with tape or markings. If the height of the camera 9 can change depending on the installation location situation, etc., it is possible to measure the height from the ground and install it so that it is at the same height, or to use the same tripod or monopod so that it is at the same height (Method 1). Alternatively, the shooting position and the direction of the 360-degree camera may be written on a map or drawing of the site, and the 360-degree camera may be installed based on that (Method 2). Alternatively, it is possible to record the position of the 360-degree camera using a high-precision GNSS (Global Navigation Satellite System) (Method 3). Alternatively, it is possible to use a SLAM (Simultaneous Localization And Mapping) technology to estimate the self-position of the camera 9. The self-position of the camera 9 can also be estimated by image processing technology from markers set as landmarks on the site or the shapes (feature points) of characteristic structures (so-called Visual SLAM). In combination with a camera, it is also possible to estimate the camera's self-position using a LiDAR or depth sensor (Method 4). Another possible method is to overlay a past image P' in a semi-transparent state on the current image P, and move the camera to the point where the two images most closely match while checking the difference between them (Method 5).

[0056] In the above methods 1, 2, and 3, the difference in image quality and on-site conditions between the camera 9 used in the past and the camera 9 used currently can be ignored, and the past additional information 10 can be composited and displayed on the current image as if the past and current images were taken from the same viewpoint. In the above method 4, when using a technique such as SLAM, the composite display is possible in the same way as in methods 1 to 3, unless there is a large difference in image quality and on-site conditions and few matching points. In the above method 5, even if the on-site conditions have changed significantly, the composite display is possible in the same way as in methods 1 to 4, as long as the human eye can judge and the camera 9 can be moved to the same position.

[0057] When the current position and orientation of the camera 9 can be estimated by any of the above methods 1 to 5, it becomes possible to automatically and correctly overlay the past additional information 10 onto the current video P. For example, it is possible to adopt a mechanism that records position information (coordinates, altitude, direction) for the past video P as well, and uses the position information of the current video P as a reference to perform calculations so that the coordinates and direction of the past video P are the same, thereby determining the composite display position.

[0058] When there are multiple pieces of additional information 10 created in the past, the additional information 10 may be selected in chronological order, such as the latest one, or the participants may select any one or more pieces of additional information 10 from a list of additional information 10 and synthesize them. When synthesizing multiple pieces of additional information 10, they may be synthesized together, or the additional information 10 may be automatically switched and synthesized at regular intervals so that the synthesized additional information 10 changes, for example, in chronological order or in an arbitrary order. In addition, a system may be constructed in which additional information 10 that is estimated to be suitable based on tag information attached to the 360-degree video P and the additional information 10, the contents of the electronic whiteboard, construction plan information, etc. is presented as candidates, and the participants may select from there. In this case, it is possible to use AI or suggestions based on weighting.

[0059] It is also possible to further link related information to the stored additional information 10. For example, by linking electronic data of construction documents (architectural drawings, construction plans, work standards, inspection item lists, schedules, photo ledgers, etc.), it is possible to display and refer to them together with the additional information 10.

[0060] It is also possible to link with map information and display local map information together with the additional information 10 based on the latitude and longitude information of the current location. By simultaneously displaying map information on the screen, it becomes possible to realize a remote presence in which the participants understand the location information. The map to be displayed may be an online map service or an image including a map.

[0061] The 360-degree camera 9 used at the site may be a stationary type, but is preferably movable, for example, as long as it can be carried by the participants (workers) at the site. It is also possible to mount the 360-degree camera 9 on a drone and move the drone together with the 360-degree camera 9 by remote control using the information device A. In this case, it can be used in situations where it is necessary to check the site during a natural disaster or other disaster, but people cannot get close to the site.

[0062] Examples of witnessed inspections (step confirmation inspections) to which the remote presence support system 1 can be applied are as follows. Upon completion of installation of designated temporary construction work: Confirmation items "Materials used, width, height, length, depth, etc." Upon completion of the surface mixing process and roadbed stabilization process of the surface treatment stabilization work, upon completion of excavation for the replacement process, and upon completion of the sand mat process -When driving sheet piles and when driving is completed When driving prefabricated piles, etc. - When driving steel pipe sheet pile foundations, etc. When the normal line of embankment and revetment works is completed - Cylinder box construction, structural construction, RC structural construction, pier footing construction, RC retaining wall construction, erosion control dam, weir main construction, pumping station main construction, sluice gate construction, utility conduit main construction (important structures): items to be checked when rebar assembly is completed (materials used, comparison with design drawings, number of spacers); items to be checked when backfilling (comparison with design drawings (finished form of invisible parts)); items to be understood when pouring concrete (quality standards, transportation time, pouring sequence, weather, temperature) - When each layer of paving work, including the roadbed, base layer, and surface layer, is completed and when paving is completed Upon completion of road cutting work During construction and completion of shaft construction (sewerage) Completion of steel bar assembly for manhole construction (sewerage) etc. - When laying and compacting earthworks (embankments)

[0063] The remote presence support system 1 can also be used in the following situations, for example. · Real estate property viewing · Sales proposals Inspection at the time of moving in and moving out of the property Inspection by management companies and construction companies before and after work - When inspecting social infrastructure (bridges and tunnels) -Remote presence of damage situation during disasters · Remote assessment of accident and disaster sites for insurance companies · Home inspection by a home inspector

[0064] Needless to say, the modified examples given in this specification may be combined as appropriate. [Explanation of symbols]

[0065] 1. Remote presence support system 2 Network 3 CPU 4. Communications Department 5 Input section 6 Output section 7 Memory section 8 Bus 9. 360-degree camera 10 Additional Images 10a Handwritten lines 10b Electronic small blackboard A Information equipment (A1~A3) M mark (M1~M3) P 360 degree video S Web conference screen

Claims

1. A remote presence support system that controls communication between a plurality of information devices to enable a web conference to be held while viewing a 360-degree image captured by an imaging device at a site to be remotely supported, thereby enabling remote support through a web conference, a web conference screen display means for displaying a web conference screen including the 360-degree video on an output unit of the plurality of information devices; A storage means for storing the 360-degree video; A remote presence support system having an instruction information sharing means for, when instruction information is created by adding additional information to the 360-degree video stored in the storage means using one of the plurality of information devices and an instruction information sharing instruction for sharing the instruction information is received from the information device, outputting the instruction information to an output unit of the information device related to the instruction information sharing instruction.

2. The storage means stores the 360-degree image and the additional information constituting the instruction information in a mutually linked state, 2. The remote presence support system according to claim 1, further comprising a past additional information adding means for adding, upon receiving from one of the plurality of information devices, an instruction to add additional information linked to a past 360-degree image captured at the same position to any 360-degree image used for remote presence, the additional information stored in the storage means to the any 360-degree image.

3. 3. The remote presence support system according to claim 1, further comprising a past video sharing means for, when receiving a past video sharing instruction from one of the plurality of information devices to share a specific past 360-degree video stored in the storage means, including the specific past 360-degree video on a web conference screen of the information device related to the past video sharing instruction.

4. The remote presence support system according to claim 3 , wherein at least one of the plurality of information devices is a head mounted display.

5. The remote presence support system according to claim 4, further comprising a thermography overlay display means for overlaying a thermography obtained by measuring the surrounding temperature using a thermography camera placed at the shooting point of the imaging device at the site on a 360-degree image.

6. The remote presence support system according to claim 5, further comprising a three-dimensional data synthesis means for superimposing three-dimensional data obtained by three-dimensionally scanning the surroundings using a LiDAR sensor disposed at the shooting point of the imaging device at the site onto a 360-degree image.

Images