Remote support system, terminal device, remote device, guidance image display program, and remote support program

Abstract

To provide a remote support system, a terminal device, a remote device, a guide image display program, and a remote support program capable of guiding a user to a requested photographing position and photographing posture more accurately compared with a case where a photographing direction is not designated.SOLUTION: A terminal device 30 transmits to a remote device 10 image information obtained by photographing a space including an object, receives display information indicating a display position and posture of a guide image virtually displayed in a three-dimensional space in a photographing region to the object, the guide image being derived by the remote device 10 depending on the image information and three-dimensional position information of a body included in the space, and displays the guide image by using the received display information. The remote device 10 receives the image information from the terminal device 30, derives the display information depending on the received image information and the three-dimensional position information, and transmits the derived display information to the terminal device 30.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a remote assistance system, a terminal device, a remote device, a guidance image display program, and a remote assistance program. [Background technology]

[0002] Patent Document 1 discloses a guidance support method aimed at smoothly guiding a user to a target. This guidance support method detects target information from pre-stored spatial information, and measures the position of a display device in a space defined by the spatial information and the position of a target defined by the target information. This guidance support method also determines the position and movement of an icon to be displayed on the display device based on the measured positions of the display device and the target, and displays the icon at the determined position and with the determined movement on the display device. This guidance support method then performs a process of changing the position and movement of the icon displayed on the display device in response to changes in the position of the display device.

[0003] Patent Document 2 also discloses an information processing system that controls the display of information input to a receiving device for captured images. This information processing system includes a control unit that controls the display of an aggregate image obtained by aggregating first information generated in response to an input to a first system that receives captured images by an imaging system and second information generated in response to an input to a second system that receives captured images by the imaging system.

[0004] Furthermore, Patent Document 3 discloses a remote work support device including a site terminal having a camera unit that captures an image seen from the worker's perspective and an instruction terminal that transmits and receives information to and from the site terminal. In this remote work support device, the instruction terminal includes a position and direction estimation unit that estimates the position and orientation of the worker from the image captured by the camera unit, and a site situation image generation unit that generates an image showing the site situation including the worker's position from the estimation result by the position and direction estimation unit. Furthermore, in this remote work support device, the instruction terminal includes an instruction-side display unit that displays a screen including the image generated by the site situation image generation unit, a work instruction receiving unit that receives information indicating the next work position input by a work instructor on the screen displayed by the instruction-side display unit, and a direction calculation unit that calculates the direction to the next work position from the estimation result by the position and direction estimation unit and the reception result by the work instruction receiving unit. The remote work support device is provided with a guide image generation unit in which the on-site terminal generates an image indicating the direction to the next work position from the calculation results by the direction calculation unit, and a on-site display unit that displays a screen including the image generated by the guide image generation unit. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-068689 [Patent Document 2] International Publication No. 2017 / 056631 [Patent Document 3] International Publication No. 2017 / 158718 Summary of the Invention [Problem to be solved by the invention]

[0006] The present invention aims to provide a remote assistance system, terminal device, remote device, guidance image display program, and remote assistance program that can more accurately guide a user to the required shooting position and shooting posture compared to when the shooting direction is not specified. [Means for solving the problem]

[0007] In order to achieve the above-mentioned object, a remote assistance system according to a first aspect is a remote assistance system including a terminal device and a remote device, wherein the terminal device is provided with a first processor, the first processor transmits image information obtained by photographing a space including an object to the remote device, receives display information indicating a display position and posture of a guidance image virtually displayed in a three-dimensional space of a photographing area for the object, the display information being derived by the remote device according to the image information and three-dimensional position information of an object included in the space, and displays the guidance image using the received display information, and the remote device is provided with a second processor, the second processor receives the image information from the terminal device, derives the display information according to the received image information and the three-dimensional position information, and transmits the derived display information to the terminal device. Here, the three-dimensional position information is information indicating the three-dimensional position of an object contained in a space corresponding to the imaging angle of view of the imaging.

[0008] In addition, a remote support system according to a second aspect is the remote support system according to the first aspect, wherein the first processor converts the three-dimensional position information into the Remote Device The information is then transmitted to the

[0009] Furthermore, the remote support system according to the third aspect is a remote support system according to the first or second aspect, in which the guidance image has an instruction surface that is virtually displayed in the posture at the display position in the three-dimensional space.

[0010] A remote support system according to a fourth aspect is the remote support system according to the third aspect, wherein the guidance image is a polyhedron having the pointing surface.

[0011] A remote support system according to a fifth aspect is the remote support system according to the fourth aspect, wherein the polyhedron is a regular polyhedron.

[0012] Furthermore, the remote support system according to the sixth aspect is a remote support system according to the fourth or fifth aspect, wherein the instruction surface is a surface inside the polyhedron and has vertices that are some of the vertices of the polyhedron.

[0013] In addition, a remote support system according to a seventh aspect is a remote support system according to any one of the fourth to sixth aspects, in which the first processor, in addition to the guidance image, displays a plurality of polyhedrons that have the same shape and dimensions as the polyhedron but do not have the instruction surface, side by side in an area including the object.

[0014] In addition, the remote support system according to the eighth aspect is a remote support system according to any one of the fourth to seventh aspects, in which the first processor further displays a mark whose shape and dimensions match those of the instruction surface when the actual shooting state corresponds to the display position and the posture.

[0015] Furthermore, a remote support system according to a ninth aspect is a remote support system according to any one of the fourth to eighth aspects, wherein the polyhedron includes the indication surface and further includes a columnar image leading to the target object.

[0016] A remote support system according to a tenth aspect is the remote support system according to the ninth aspect, wherein the columnar image is a cylindrical image.

[0017] In addition, a remote support system according to an eleventh aspect is a remote support system according to any one of the first to tenth aspects, in which the first processor changes the display state of the guidance image according to the amount of deviation between the actual shooting state and the shooting state corresponding to the display position and the posture.

[0018] A remote support system according to a twelfth aspect is the remote support system according to the eleventh aspect, wherein the display state is at least one of a blinking interval, a display color, and a transmittance.

[0019] In addition, a remote support system according to a thirteenth aspect is a remote support system according to the twelfth aspect, in which, when the first processor changes the transmittance, the transmittance is increased as the amount of deviation becomes smaller.

[0020] In addition, a remote support system according to a 14th aspect is a remote support system according to any one of the 1st to 13th aspects, in which the first processor virtually displays a marker at a target position of an object when the target position is photographed.

[0021] In order to achieve the above object, a terminal device according to a fifteenth aspect includes a processor, and the processor transmits image information obtained by photographing a space including an object to a remote device, and the image information , and three-dimensional position information of objects contained in the space The remote device receives display information indicating the display position and posture of a guidance image that is virtually displayed in the three-dimensional space of the shooting area for the object, which is derived by the remote device in response to the received display information, and displays the guidance image using the received display information. Here, the three-dimensional position information is information indicating the three-dimensional position of an object contained in a space corresponding to the imaging angle of view of the imaging.

[0022] In order to achieve the above object, a remote device according to a sixteenth aspect includes a processor, and the processor receives image information obtained by photographing a space including an object from a terminal device, and the received image information , and three-dimensional position information of objects contained in the space In response to the request, display information indicating the display position and orientation of a guidance image that is virtually displayed in the three-dimensional space of the shooting area for the object is derived, and the derived display information is transmitted to the terminal device. Here, the three-dimensional position information is information indicating the three-dimensional position of an object contained in a space corresponding to the imaging angle of view of the imaging.

[0023] In order to achieve the above object, a guidance image display program according to a seventeenth aspect of the present invention includes: transmitting image information obtained by photographing a space including an object to a remote device; , and three-dimensional position information of objects contained in the space The computer executes a process of receiving display information indicating the display position and posture of a guidance image that is virtually displayed in the three-dimensional space of the shooting area for the object, which is derived by the remote device in response to the received display information, and displaying the guidance image using the received display information. Here, the three-dimensional position information is information indicating the three-dimensional position of an object contained in a space corresponding to the imaging angle of view of the imaging.

[0024] Furthermore, in order to achieve the above object, a remote assistance program according to an eighteenth aspect of the present invention is a program for receiving image information obtained by photographing a space including an object from a terminal device, and , and three-dimensional position information of objects contained in the space In response to the request, the computer is caused to execute a process of deriving display information indicating the display position and posture of a guidance image that is virtually displayed in the three-dimensional space of the shooting area for the object, and transmitting the derived display information to the terminal device. Here, the three-dimensional position information is information indicating the three-dimensional position of an object contained in a space corresponding to the imaging angle of view of the imaging. [Effects of the Invention]

[0025] According to the first and fifteenth to eighteenth aspects, the user can be more accurately guided to the required photographing position and posture, compared to when no photographing direction is specified.

[0026] According to the second aspect, it is possible to register three-dimensional position information with higher accuracy in the remote device compared to when the three-dimensional position information is registered in advance in the remote device.

[0027] According to the third aspect, the user can be more intuitively guided to the required photographing position and posture, compared to when the guidance image does not have an instruction surface.

[0028] According to the fourth aspect, compared to when the guidance image is not a polyhedron, the user can be more intuitively guided to the required shooting position and shooting posture.

[0029] According to the fifth aspect, it is possible to more easily guide the user to the required photographing position and posture, compared to when the polyhedron is not a regular polyhedron.

[0030] According to the sixth aspect, the indication plane can be set by specifying the vertices of the polyhedron, and as a result, the guidance image can be set more easily compared to when the vertices of the polyhedron are not used as the indication plane.

[0031] According to the seventh aspect, compared to when multiple polyhedrons without indication surfaces are not displayed, it is easier to grasp the shooting position and shooting direction indicated by the guidance image even when the target object is located relatively far away.

[0032] According to the eighth aspect, it is possible to more easily guide the user to the required photographing position and posture, compared to when no mark is displayed.

[0033] According to the ninth aspect, the sense of distance to the object can be improved compared to when columnar images are not included, and as a result, the user can be more easily guided to the required shooting position and posture.

[0034] According to the tenth aspect, the sense of distance to the object can be further enhanced compared to when a columnar image is used as a prismatic image.

[0035] According to the eleventh aspect, the user can be guided to the required shooting position and shooting posture in a shorter time than when the display state of the guidance image is not changed according to the deviation between the actual shooting state and the shooting state corresponding to the above-mentioned display position and posture.

[0036] According to the twelfth aspect, it is possible to grasp the amount of deviation between the actual shooting state and the required shooting state by utilizing the applied change in the display state.

[0037] According to the thirteenth aspect, when the transmittance is changed, the visibility of the object can be improved as the amount of deviation increases, compared to when the transmittance is increased.

[0038] According to the fourteenth aspect, it is possible to allow the user to more easily grasp the target position of the object, compared to when a marker is not displayed at the target position of the object. [Brief explanation of the drawings]

[0039] [Figure 1]FIG. 2 is a block diagram illustrating an example of a hardware configuration of the remote support system according to the embodiment. [Figure 2] FIG. 2 is a block diagram illustrating an example of a functional configuration of the remote support system according to the embodiment. [Figure 3] FIG. 10 is a perspective view illustrating an example of a guidance image according to the embodiment. [Figure 4] 10 is a flowchart illustrating an example of a guidance image display process according to the embodiment. [Figure 5] FIG. 10 is a front view showing an example of a guidance image display screen according to the embodiment. [Figure 6] FIG. 10 is a front view showing another example of a guidance image display screen according to the embodiment. [Figure 7] 10 is a flowchart illustrating an example of a remote assistance process according to the embodiment. [Figure 8] FIG. 10 is a front view showing an example of a guidance image setting screen according to the embodiment. [Figure 9] FIG. 10 is a front view showing another example of a guidance image setting screen according to the embodiment. [Figure 10] FIG. 10 is a perspective view showing another example of a guidance image according to the embodiment. [Figure 11] FIG. 10 is a perspective view showing another example of a guidance image according to the embodiment. [Figure 12] FIG. 10 is a perspective view showing another example of a guidance image according to the embodiment. [Figure 13] FIG. 10 is a perspective view showing another example of a guidance image according to the embodiment. [Figure 14] FIG. 10 is a front view showing another example of a guidance image display screen according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0040] Hereinafter, with reference to the drawings, an example of an embodiment of the present invention will be described in detail. In this embodiment, the present invention will be described as being applied to a remote support system including a plurality of terminal devices, each of which is a portable terminal used individually by a user, and a remote device installed in a remote location away from each of the terminal devices. In addition, in this embodiment, the users are workers who perform maintenance on image forming devices such as digital multifunction peripherals and printers, and the remote support system remotely supports the workers in performing maintenance on the image forming devices.

[0041] First, the configuration of a remote support system 90 according to this embodiment will be described with reference to Fig. 1 and Fig. 2. Fig. 1 is a block diagram showing an example of the hardware configuration of the remote support system 90 according to this embodiment. Fig. 2 is a block diagram showing an example of the functional configuration of the remote support system 90 according to this embodiment.

[0042] 1, a remote support system 90 according to this embodiment includes a remote device 10 and multiple terminal devices 30, each of which is capable of accessing a network 80. Examples of the remote device 10 include information processing devices such as a personal computer and a server computer. Examples of the terminal devices 30 include portable terminals such as a smartphone, a tablet terminal, and a PDA (Personal Digital Assistant, mobile information terminal).

[0043] The terminal device 30 according to this embodiment is a device carried by each of multiple users (hereinafter simply referred to as "users") who use the remote support system 90 when performing maintenance on the image forming device. The terminal device 30 includes a CPU (Central Processing Unit) 31, a memory 32 serving as a temporary storage area, a nonvolatile memory unit 33, an input unit 34 such as a touch panel, a display unit 35 such as a liquid crystal display, and a medium read / write device (R / W) 36. The terminal device 30 also includes an image capture unit 38, a microphone 39, a position detection unit 40, a spatial information detection unit 41, and a wireless communication unit 42. The CPU 31, memory 32, memory unit 33, input unit 34, display unit 35, medium read / write device 36, image capture unit 38, microphone 39, position detection unit 40, spatial information detection unit 41, and wireless communication unit 42 are interconnected via a bus B1. The medium read / write device 36 reads information from and writes information to a recording medium 37.

[0044] The storage unit 33 is realized by a hard disk drive (HDD), a solid state drive (SSD), a flash memory, or the like. A guidance image display program 33A is stored in the storage unit 33 as a storage medium. The guidance image display program 33A is stored in the storage unit 33 when a recording medium 37 having the guidance image display program 33A written therein is set in the medium reading and writing device 36 and the medium reading and writing device 36 reads the guidance image display program 33A from the recording medium 37. The CPU 31 reads the guidance image display program 33A from the storage unit 33, expands it in the memory 32, and sequentially executes processes included in the guidance image display program 33A.

[0045] The photographing unit 38 according to the present embodiment photographs the image forming apparatus to be maintained and outputs image information obtained by the photographing. The microphone 39 according to the present embodiment collects voices uttered by the user and outputs voice information.

[0046] Furthermore, the position detection unit 40 according to this embodiment detects its own position and outputs position information, and although a GPS (Global Positioning Systems) is used in this embodiment, the present invention is not limited to this. For example, the position detection unit 40 may use position information acquired from a Wi-Fi (registered trademark) router, detect the position using a beacon, or detect the position by image analysis using a captured image.

[0047] Furthermore, the spatial information detection unit 41 according to this embodiment detects information indicating the three-dimensional position of an object contained in the space in front of the terminal device 30 (hereinafter referred to as "three-dimensional position information"). The spatial information detection unit 41 according to this embodiment detects three-dimensional position information of the space corresponding to the angle of view of the image captured by the image capture unit 38, and in this embodiment, a depth sensor is used as the spatial information detection unit 41, but this is not limiting. For example, a three-dimensional scanner or the like may be used as the spatial information detection unit 41.

[0048] On the other hand, the remote device 10 plays a central role in the remote assistance system 90, and is a device that derives a guidance image to be presented to the user, which will be described in detail later. The remote device 10 includes a CPU 11, a memory 12 as a temporary storage area, a non-volatile memory unit 13, an input unit 14 such as a keyboard and a mouse, a display unit 15 such as a liquid crystal display, a medium reading / writing device 16, and a communication interface (I / F) unit 18. The CPU 11, memory 12, memory unit 13, input unit 14, display unit 15, medium reading / writing device 16, and communication I / F unit 18 are connected to one another via a bus B2. The medium reading / writing device 16 reads information written in a recording medium 17 and writes information to the recording medium 17.

[0049] The storage unit 13 is realized by an HDD, an SSD, a flash memory, or the like. A remote assistance program 13A is stored in the storage unit 13 as a storage medium. The remote assistance program 13A is stored in the storage unit 13 when a recording medium 17 on which the remote assistance program 13A is written is set in the medium reading and writing device 16 and the medium reading and writing device 16 reads the remote assistance program 13A from the recording medium 17. The CPU 11 reads the remote assistance program 13A from the storage unit 13, expands it in the memory 12, and sequentially executes the processes of the remote assistance program 13A.

[0050] In this embodiment, the network 80 is a public communication line such as the Internet or a telephone network, but is not limited to this. For example, the network 80 may be an in-company communication line such as a LAN (Local Area Network) or a WAN (Wide Area Network), or may be a combination of these in-company communication lines and public communication lines.

[0051] Next, the functional configuration of the remote device 10 and the terminal device 30 according to this embodiment will be described with reference to FIG.

[0052] 2, the terminal device 30 according to this embodiment includes a first transmitter 31A, a first receiver 31B, and a first display controller 31C. The CPU 31 of the terminal device 30 executes a guidance image display program 33A, thereby functioning as the first transmitter 31A, the first receiver 31B, and the first display controller 31C.

[0053] The first transmission unit 31A according to this embodiment transmits image information (hereinafter simply referred to as "image information") obtained by photographing a space including an object to the remote device 10. As described above, in this embodiment, the object is an image forming device that is the target of maintenance.

[0054] Furthermore, the first receiving unit 31B according to this embodiment receives image information and display information (hereinafter simply referred to as "display information") indicating the display position and posture of a guidance image virtually displayed in the three-dimensional space of the shooting area for the target, which is derived by the remote device 10 according to three-dimensional position information of the object included in the space. Note that, in this embodiment, the first transmitting unit 31A transmits the image information and the first receiving unit 31B receives the display information via the wireless communication unit 42, but this is not limiting.

[0055] Then, the first display control unit 31C according to the present embodiment displays a guidance image on the display unit 35 using the received display information.

[0056] 2, the remote device 10 according to this embodiment includes a second receiving unit 11A, a second display control unit 11B, a derivation unit 11C, and a second transmitting unit 11D. The CPU 11 of the remote device 10 executes the remote assistance program 13A, thereby functioning as the second receiving unit 11A, the second display control unit 11B, the derivation unit 11C, and the second transmitting unit 11D.

[0057] The second receiving unit 11A according to this embodiment receives image information from the terminal device 30. The derivation unit 11C according to this embodiment derives the display information according to the received image information and the above-mentioned three-dimensional position information. The second transmitting unit 11D according to this embodiment transmits the derived display information to the terminal device 30.

[0058] In the remote assistance system 90 according to this embodiment, the second display control unit 11B uses the received image information and the above-mentioned three-dimensional position information to display a guidance image setting screen, the details of which will be described later, on the display unit 15. Then, the derivation unit 11C according to this embodiment derives display information by having the user of the remote device 10 set the display position and posture of the guidance image on the guidance image setting screen displayed by the display unit 15.

[0059] Furthermore, in this embodiment, image information indicating a still image is applied as the image information. Therefore, the photographing unit 38 provided in the terminal device 30 is assumed to be capable of photographing still images, but this is not limited thereto. For example, it goes without saying that a photographing unit capable of photographing both still images and moving images may be applied as the photographing unit 38. Furthermore, in this embodiment, image information indicating a color image is applied as the image information, and therefore the photographing unit 38 according to this embodiment is assumed to be capable of photographing color images, but this is not limited thereto. For example, image information indicating a monochrome image may be applied as the image information, and the photographing unit 38 may be configured to be capable of photographing monochrome images.

[0060] In addition, in this embodiment, the three-dimensional position information used by the remote device 10 is obtained by the spatial information detection unit 41 of the terminal device 30, transmitted by the first transmission unit 31A, and received by the second reception unit 11A of the remote device 10. However, this is not limited to this form, and three-dimensional position information obtained separately using a measuring device different from the spatial information detection unit 41 of the terminal device 30 may be stored in advance in the storage unit 13 of the remote device 10.

[0061] In addition, in this embodiment, the guidance image has an indication surface that is virtually displayed at the display position in the three-dimensional space in the above-mentioned attitude, and in particular, the guidance image according to this embodiment is a polyhedron having the indication surface. Furthermore, the polyhedron according to this embodiment is a regular polyhedron. Figure 3 shows a perspective view of an example of a guidance image 60 according to this embodiment.

[0062] As shown in FIG. 3, the guidance image 60 according to this embodiment is a polyhedron 60B having the above-described indication surface 60A, and the polyhedron 60B is a regular hexahedron.

[0063] The indication surface 60A of the guidance image 60 according to this embodiment is a surface inside the polyhedron 60B, and has vertices that are some of the vertices 60a to 60h of the polyhedron 60B (vertices 60a, 60c, and 60f in the example shown in FIG. 3). Therefore, in this case, when the indication surface 60A is viewed from a direction directly facing it, the indication surface 60A has the shape of an equilateral triangle.

[0064] In this manner, in the present embodiment, the guidance image includes the polyhedron 60B, but the present invention is not limited to this. For example, the guidance image may be only the indication surface 60A without the polyhedron 60B.

[0065] Furthermore, the first display control unit 31C according to the present embodiment changes the display state of the guidance image 60 according to the amount of deviation between the actual photographing state of the terminal device 30 and the photographing state corresponding to the display position and posture. Here, in the present embodiment, transmittance is applied as the display state, and the transmittance is increased as the amount of deviation decreases. However, this is not limited to this embodiment, and instead of the transmittance, either one of the blinking interval and the display color may be applied as the display state, or a combination of two or three of the blinking interval, the display color, and the transmittance may be applied as the display state.

[0066] Furthermore, in this embodiment, when a target position of the object is photographed, the first display control unit 31C virtually displays a marker at that position.

[0067] Next, the operation of the remote support system 90 according to this embodiment will be described with reference to Figs. 4 to 9. First, the operation of the terminal device 30 according to this embodiment will be described with reference to Figs. 4 to 6. Fig. 4 is a flowchart showing an example of a guidance image display process according to this embodiment. In order to avoid confusion, the following description will omit a description of the process for giving instructions regarding the work itself that is the target of remote support by the remote support system 90, and will only describe a process for guiding the user to the required shooting position and shooting posture.

[0068] In the remote support system 90 according to this embodiment, a user uses a terminal device 30 to photograph a space including an object from a predetermined direction (in this embodiment, the front of the object), and the image information obtained by the photographing unit 38 and the three-dimensional position information obtained by the spatial information detection unit 41 at the same time as the photographing are stored in a memory unit 33.

[0069] In this state, when the user inputs an instruction to execute the guidance image display process to the terminal device 30, the CPU 31 of the terminal device 30 executes the guidance image display program 33A, thereby executing the guidance image display process shown in Figure 4.

[0070] In step 100 of FIG. 4, the CPU 31 reads out the image information and the three-dimensional position information from the storage unit 33, and in step 102, the CPU 31 transmits the acquired image information and the three-dimensional position information to the remote device .

[0071] Although details will be described later, upon receiving the image information and the three-dimensional position information, the remote device 10 uses this information to derive the above-mentioned display information (hereinafter referred to as "guidance image display information"), which is information for displaying a guidance image 60 for guiding the user to a required photographing position and photographing posture. Then, the remote device 10 transmits the derived guidance image display information to the accessing terminal device 30. At this time, the remote device 10 transmits, together with the guidance image display information, marker display information for virtually displaying a marker at a position (hereinafter referred to as a "target") that is a target for maintenance of the object to the terminal device 30.

[0072] Therefore, in step 104, the CPU 31 waits until guidance image display information and marker display information are received from the remote device 10.

[0073] In step 106, the CPU 31 identifies its own photographing position and attitude (same as the photographing direction) at this time. Here, the information indicating the photographing position is obtained by the position detection unit 40. Furthermore, the information indicating the attitude is obtained by a gyro sensor (not shown) built into the terminal device 30. However, this is not limiting, and for example, the information indicating the attitude may be obtained by an acceleration sensor or the like instead of the gyro sensor.

[0074] In step 108, the CPU 31 controls the display unit 35 to display a guidance image display screen having a predetermined configuration using the received guidance image display information and marker display information, the identified photographing position and posture of the device itself, and an image photographed by the photographing unit 38 at this time (hereinafter referred to as an "actual photographed image"). Fig. 5 shows an example of a guidance image display screen according to this embodiment.

[0075] 5, on the guidance image display screen according to the present embodiment, a message is displayed prompting the user to set the position and orientation of the terminal device 30 so that the terminal device 30 faces the indication surface 60A in the polyhedron 60B in the guidance image 60. Furthermore, on the guidance image display screen according to the present embodiment, the guidance image 60 is displayed superimposed on the actually captured image, and the indication surface 60A is displayed at the display position and orientation indicated by the guidance image display information. Furthermore, on the guidance image display screen according to the present embodiment, the display state (transmittance in the present embodiment) of the guidance image 60 is sequentially changed (in the present embodiment, the transmittance increases as the amount of deviation decreases) according to the amount of deviation between the actual captured state and the captured state corresponding to the required captured position and captured orientation.

[0076] That is, the guidance image 60 according to this embodiment is a three-dimensional image in which the indication surface 60A forms an equilateral triangle when the image capturing position and posture of the terminal device 30 are the required image capturing position and posture.

[0077] In the terminal device 30 according to this embodiment, when the shooting position and posture of the terminal device 30 are different from the required shooting position and posture, the guidance image 60, in which the indication surface 60A appears as an equilateral triangle when the terminal device 30 is in the required shooting position and posture, is displayed at the shooting position.

[0078] 5 shows an example of a guidance image display screen in the case where the target object 50 is currently being photographed from the front and the requested photographing position is the left side of the target object 50. In this case, the instruction surface 60A of the guidance image 60 is a distorted equilateral triangle, and its size becomes smaller as the distance from the terminal device 30 to the requested photographing position increases. Therefore, by referring to the guidance image display screen, the user can intuitively grasp the photographing position and posture requested for the target object 50 from the display position and size of the guidance image 60 and the shape of the instruction surface 60A.

[0079] The example shown in Fig. 6 is an example of a guidance image display screen when the terminal device 30 approaches the required shooting position and posture in the example shown in Fig. 5. In this case, the object 50 is displayed as if it were shot from the left side, and the indication surface 60A of the guidance image 60 resembles an equilateral triangle. In this case, the target is included in the shooting angle of view, so the above-mentioned marker 70 is displayed. This allows the user to easily grasp the position of the target.

[0080] In the present embodiment, the terminal device 30 sequentially generates information indicating the guidance image 60 having different display positions, shapes, sizes, and transmittances depending on the shooting position and posture of the terminal device 30. However, the present invention is not limited to this. For example, the remote device 10 may sequentially generate information indicating the guidance image 60 and transmit it to the terminal device 30.

[0081] In step 110, the CPU 31 determines whether the terminal device 30 has reached the required shooting position and posture and has completed guiding the user to the destination, and if the determination is negative, the process returns to step 106, whereas if the determination is positive, the CPU 31 terminates this guidance image display process.

[0082] By repeating the process of steps 106 to 110, the display unit 35 of the terminal device 30 displays the object 50 being photographed by the terminal device 30, and also displays a superimposed guidance image 60 in a state corresponding to the photographing position and posture of the terminal device 30. Therefore, by referring to this display screen, the user can intuitively set the terminal device 30 to the required photographing position and posture.

[0083] When the guidance image display process is completed, a remote support process for performing necessary maintenance on the object 50 is executed using a conventionally known technique.

[0084] Next, the operation of the remote device 10 according to this embodiment will be described with reference to Figures 7 to 9. Figure 7 is a flowchart showing an example of the remote assistance process according to this embodiment. Note that the case where the user of the remote device 10 is a person who gives remote instructions to the user (hereinafter referred to as "remote instructor") will be described.

[0085] In the remote support system 90 according to this embodiment, the CPU 11 of the remote device 10 starts executing the remote support program 13A every predetermined period (10 minutes in this embodiment) during the available time period (8:00 AM to 5:00 PM in this embodiment) on the day (weekday in this embodiment) when the remote support system 90 is available to the user, thereby executing the remote support processing shown in FIG. 7.

[0086] 7, the CPU 11 waits until image information and three-dimensional position information are received from any of the terminal devices 30. In step 202, the CPU 11 controls the display unit 15 to display a guidance image setting screen having a predetermined configuration using the received information, and then in step 204, the CPU 11 waits until the predetermined information is input. Fig. 8 shows an example of the guidance image setting screen according to this embodiment.

[0087] 8, the guidance image setting screen according to this embodiment displays a message prompting the user to specify the position of the polyhedron and the pointing surface, and a message prompting the user to specify the position of the target described above. Furthermore, the guidance image setting screen according to this embodiment virtually displays the three-dimensional space of the captured area corresponding to the captured image, including the target object 50, using the captured image indicated by the received image information and the three-dimensional positions of each object indicated by the received three-dimensional position information. Furthermore, the guidance image setting screen according to this embodiment displays the polyhedron 60B in the guidance image 60 at a predetermined default position and with a default size.

[0088] Therefore, by referring to the guidance image setting screen, the remote instructor can grasp the position of the object 50 photographed by the accessing user (hereinafter simply referred to as the "user") using the terminal device 30 as a position in the three-dimensional space. Then, the remote instructor moves the displayed polyhedron 60B to a requested photographing position (in this embodiment, the position of the user at the time of starting maintenance of the object 50) via the input unit 14. At this time, the CPU 11 of the remote device 10 successively changes the size of the polyhedron 60B depending on the distance in the three-dimensional space between the position of the polyhedron 60B being moved by the remote instructor and the photographing position by the terminal device 30. Therefore, the remote instructor can grasp the position of the polyhedron 60B in the forward / backward direction based on the size of the polyhedron 60B.

[0089] When the movement of the polyhedron 60B to the requested photographing position is completed, the remote instructor specifies, via the input unit 14, the vertices of the polyhedron 60B that constitute the instruction surface that will be oriented in the requested photographing direction or in the direction closest to the photographing direction (tilt angle). In response to this specification, the guidance image 60 is displayed on the guidance image setting screen in the state shown in FIG. 9 as an example.

[0090] In this manner, in the present embodiment, the operation related to the guidance image 60 is performed by moving the polyhedron 60B to the required photographing position and then specifying the instruction plane 60A, but this is not limitative. For example, the guidance image 60 may be moved after specifying the instruction plane 60A.

[0091] The remote instructor also specifies the position of the target on the object 50 via the input unit 14. At this time, the target position on the object 50 may not be displayed on the displayed guidance image setting screen. In this case, the remote instructor rotates the object 50 or the guidance image setting screen itself in three-dimensional space so that the target position can be seen, and then specifies the target position.

[0092] Then, when the remote instructor has finished specifying the guidance image 60 and the target position, he or she presses the end button 15A via the input unit 14. In response to this, the determination in step 204 is affirmative, and the process proceeds to step 206.

[0093] In step 206, the CPU 11 derives the above-mentioned guidance image display information and marker display information using various information specified by the remote instructor on the guidance image setting screen. In step 208, the CPU 11 transmits the derived guidance image display information and marker display information to the accessing terminal device 30, and then terminates the remote assistance process.

[0094] In the above embodiment, the case where transmittance is used as the display state of the guidance image that changes depending on the amount of deviation between the actual shooting state and the shooting state corresponding to the required shooting position and shooting posture has been described, but this is not limited thereto. For example, as described above, at least one of the blinking interval and the display color may be used as the display state, or a combination of multiple types of three display states, which are the blinking interval, the display color, and the transmittance, may be used as the display state. For example, when the blinking interval is used as the display state, an example is exemplified in which the blinking interval is shortened as the amount of deviation decreases, and when the display color is used as the display state, an example is exemplified in which the color approaches red as the amount of deviation decreases.

[0095] In the above embodiment, the instruction surface of the guidance image 60 is the instruction surface 60A inclined at 45 degrees as shown in Fig. 3, but the present invention is not limited to this. For example, a vertical surface may be used as the instruction surface 60A.

[0096] In this case, if the imaging direction is to be specified as a diagonally forward right direction, four vertices, vertices 60a, 60c, 60e, and 60g, may be specified, as shown in FIG. 10 . Furthermore, if the outer peripheral surface of polyhedron 60B is to be specified as indication surface 60A, for example, vertices 60b, 60c, 60f, and 60g may be specified. In this case, instead of specifying four vertices, the surface to be used as indication surface 60A may be directly specified. In this case, indication surface 60A can be specified with only one specifying operation, which makes it easier to specify indication surface 60A than specifying indication surface 60A by specifying vertices.

[0097] In the above embodiment, the case where a polyhedron is a regular hexahedron is described as the guidance image, but the present invention is not limited thereto. For example, as shown in FIG. 11, a guidance image 64 in which the polyhedron is a regular tetrahedron may be applied, or a hexahedron or tetrahedron with different side lengths may be applied. Furthermore, the guidance image using a polyhedron is not limited to the one having only the indication surface 60A as described above. For example, as shown in FIG. 12, a guidance image 66 including the indication surface and a columnar image 66A leading to the target may be applied. Note that, to avoid confusion, the indication surface and the target are not shown in FIG. 12.

[0098] 13, in addition to the guidance image 60, a plurality of polyhedrons 60B having the same shape and dimensions as the polyhedron 60B of the guidance image 60 but not having an instruction surface 60A may be displayed side by side on the display unit 35 of the terminal device 30 so as to encompass the area including the target object 50. According to this configuration, even if the target object 50 is located relatively far away, it is easy to grasp the required shooting position and posture designated by the remote instructor.

[0099] Although not mentioned in the above embodiment, as an example, a configuration may be adopted in which the first display control unit 31C further displays a mark 60C whose shape and dimensions match those of the indication surface 60A when the actual photographing state matches the required photographing position and photographing posture, as shown in Fig. 14. In this configuration, when the indication surface 60A and the mark 60C match, information indicating that user guidance has ended may be automatically transmitted to the remote device 10.

[0100] Although the embodiments have been described above, the technical scope of the present invention is not limited to the scope described in the above embodiments. Various modifications and improvements can be made to the above embodiments without departing from the gist of the invention, and such modifications and improvements are also included in the technical scope of the present invention.

[0101] Furthermore, the above-described embodiments do not limit the inventions described in the claims, and not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention. The above-described embodiments include inventions at various stages, and various inventions can be extracted by combining the multiple disclosed constituent elements. Even if some constituent elements are deleted from all of the constituent elements shown in the embodiments, as long as the effect is still obtained, the configuration from which these constituent elements are deleted can be extracted as an invention.

[0102] Furthermore, in the above embodiments, the term "processor" refers to a processor in a broad sense, and includes general-purpose processors (e.g., CPUs, etc.) and dedicated processors (e.g., GPU: Graphics Processing Unit, ASIC: Application Specific Integrated Circuit, FPGA: Field Programmable Gate Array, programmable logic device, etc.).

[0103] Furthermore, in the above embodiment, the guidance image display process and the remote support process are described as being implemented by a software configuration using a computer by executing a program, but the present invention is not limited to this. For example, the guidance image display process and the remote support process may be implemented by a hardware configuration or a combination of a hardware configuration and a software configuration.

[0104] Furthermore, the configurations of the remote device 10 and the terminal device 30 described in the above embodiment are merely examples, and it goes without saying that unnecessary parts may be deleted or new parts may be added within the scope of the present invention.

[0105] Furthermore, the flow of the guidance image display processing and remote support processing described in the above embodiment is also an example, and it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be changed within the scope of the gist of the present invention. [Explanation of symbols]

[0106] 10 Remote Device 11 CPU 11A Second receiving unit 11B Second display control unit 11C Derivation part 11D Second transmitter 12 Memory 13 Storage section 13A Remote Support Program 14 Input section 15 Display section 15A Exit button 16 Media reading and writing device 17 Recording Media 18 Communication I / F section 30 Terminal Equipment 31 CPU 31A First Transmitter 31B First receiving unit 31C First display control unit 32 memory 33 Storage section 33A Guidance image display program 34 Input section 35 Display section 35A Exit button 36 Media reading and writing device 37 Recording Media 38 Photography Department 39. Mike 40 Position detection unit 41 Spatial information detection unit 42 Radio Communication Department 50 Objects 60 Guided Images 60A indicator surface 60B Polyhedron 60C mark 60a~60h apex 64 Guided Image 66 Guided Image 66A Column Image 70 Marker 80 Network 90 Remote Support System

Claims

1. A remote support system including a terminal device and a remote device, the terminal device comprises a first processor; The first processor transmitting image information obtained by photographing a space including an object to the remote device; receiving display information indicating a display position and an attitude of a guidance image virtually displayed in the three-dimensional space of a shooting area for the target, the display information being derived by the remote device according to the image information and three-dimensional position information of an object included in the space; Displaying the guidance image using the received display information; the remote device comprises a second processor; The second processor receiving the image information from the terminal device; deriving the display information according to the received image information and the three-dimensional position information; transmitting the derived display information to the terminal device; It is a remote support system, The three-dimensional position information is information indicating a three-dimensional position of an object included in a space corresponding to a shooting angle of view of the shooting. Remote support system.

2. the first processor further transmits the three-dimensional position information to the remote device. The remote assistance system according to claim 1 .

3. the guidance image has an instruction surface that is virtually displayed at the display position in the three-dimensional space and in the attitude. The remote assistance system according to claim 1 or 2.

4. The guidance image is a polyhedron having the indication surface. The remote assistance system according to claim 3 .

5. The polyhedron is a regular polyhedron. The remote assistance system according to claim 4 .

6. The indication surface is a surface inside the polyhedron and has vertices that are part of the vertices of the polyhedron. The remote assistance system according to claim 4 or 5.

7. the first processor displays, in addition to the guidance image, a plurality of polyhedrons that have the same shape and dimensions as the polyhedron and do not have the indication surface, side by side in the area including the object; The remote support system according to any one of claims 4 to 6.

8. the first processor further displays a mark having the same shape and size as the indication surface when an actual photographing state corresponds to the display position and the attitude. The remote support system according to any one of claims 4 to 7.

9. The polyhedron includes the indication surface and further includes a columnar image leading to the object. The remote support system according to any one of claims 4 to 8.

10. the columnar image is a cylindrical image; The remote assistance system according to claim 9.

11. The first processor compares an actual photographing state with a photographing state corresponding to the display position and the posture. changing the display state of the guidance image according to the amount of deviation from the shadow state; The remote support system according to any one of claims 1 to 10.

12. The display state is at least one of a blinking interval, a display color, and a transmittance. The remote assistance system according to claim 11.

13. When changing the transmittance, the first processor increases the transmittance as the amount of deviation decreases. The remote assistance system according to claim 12.

14. when a target position of the object is photographed, the first processor virtually displays a marker at the target position; The remote support system according to any one of claims 1 to 13.

15. a processor; The processor: Transmitting image information obtained by photographing a space including the object to a remote device; receiving display information indicating a display position and an attitude of a guidance image virtually displayed in the three-dimensional space of a shooting area for the target, the display information being derived by the remote device according to the image information and three-dimensional position information of an object included in the space; displaying the guidance image using the received display information; a terminal device, The three-dimensional position information is information indicating a three-dimensional position of an object included in a space corresponding to a shooting angle of view of the shooting. Terminal device.

16. a processor; The processor: receiving image information obtained by photographing a space including an object from a terminal device; deriving display information indicating a display position and an orientation of a guidance image that is virtually displayed in the three-dimensional space of a shooting area relative to the target object, according to the received image information and three-dimensional position information of an object included in the space; transmitting the derived display information to the terminal device; a remote device, The three-dimensional position information is information indicating a three-dimensional position of an object included in a space corresponding to a shooting angle of view of the shooting. Remote device.

17. Transmitting image information obtained by photographing a space including the object to a remote device; receiving display information indicating a display position and an attitude of a guidance image virtually displayed in the three-dimensional space of a shooting area for the target, the display information being derived by the remote device according to the image information and three-dimensional position information of an object included in the space; displaying the guidance image using the received display information; Processing, The three-dimensional position information is information indicating a three-dimensional position of an object included in a space corresponding to a shooting angle of view of the shooting. A guidance image display program for causing a computer to execute processing.

18. receiving image information obtained by photographing a space including an object from a terminal device; deriving display information indicating a display position and an orientation of a guidance image that is virtually displayed in the three-dimensional space of a shooting area relative to the target object, according to the received image information and three-dimensional position information of an object included in the space; transmitting the derived display information to the terminal device; Processing, The three-dimensional position information is information indicating a three-dimensional position of an object included in a space corresponding to a shooting angle of view of the shooting. A remote support program that allows a computer to execute processing.

Images