Information processing device, information processing system, and information processing program

Abstract

To provide an information processing apparatus, an information processing system and an information processing program that can transmit a more clear instruction as compared with a case in which a movement quantity and a position of a superposed indication image are changed in response to each change in a distance between an imaging part and an operation object.SOLUTION: An information processing apparatus has a processor, which is configured to: acquire a video of an operation object being an object to be operated and an instruction to generate a still image from the video; generate a still image cut out of the video including the operation object as instructed; use the still image to identify a superposition region where the operation object in the video, position information being information related to a position of the operation object, and an image based upon the position of the operation object are to be superposed; receive instruction information representing an instruction for operation for the operation object; and superpose the instruction image being an image corresponding to the instruction information over the superposition region of the video and display the superposed image.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing system, and an information processing program. [Background technology]

[0002] Patent Document 1 discloses a work support system having a work support device with which a work assistant gives instructions to a worker, and a head-mounted display device worn by the worker, in which the work support device comprises a hand position photographing unit that photographs the assistant's fingers, a gesture determination unit that calculates position or movement information of the photographed assistant's fingers, a hand position information transmission unit that transmits the position or movement information of the fingers to the head-mounted display device, a video receiving unit that receives video information sent from the head-mounted display device, and a video display unit that displays the received video information, and the head-mounted display device comprises a transmissive display unit that displays a virtual image superimposed on an image of real space that is displayed transparently, a hand position information receiving unit that receives position or movement information of the assistant's fingers sent from the work support device, a virtual hand display unit that displays a virtual image of the assistant's fingers on the transmissive display unit based on the position or movement information of the assistant's fingers, and a video transmission unit that transmits the image of real space and the virtual image displayed on the transmissive display unit to the work support device as video information.

[0003] Patent Document 2 discloses a system comprising a first acquisition means for acquiring the position and orientation of the viewpoint of a first observer, a generation means for generating an image of a virtual space seen from a viewpoint having the position and orientation acquired by the first acquisition means, a first operation means used by the first observer to operate a virtual object, a second operation means used by a second observer who remotely assists the first observer in operating the virtual object to operate the virtual object, a second acquisition means for acquiring an image of real space seen from the viewpoint, and an output means for superimposing an image generated by the generation means on an image acquired by the second acquisition means and outputting the image to a head-mounted display device worn by the first observer and a head-mounted display device worn by the second observer, wherein the generation means generates an image of the virtual space that reflects the results of operations performed by the first operation means and the second operation means.

[0004] Patent Document 3 discloses a work support system comprising a head-mounted display device capable of displaying predetermined information, an imaging means capable of capturing an image in the line of sight of a worker wearing the head-mounted display device, and an information processing device, wherein the information processing device includes an instruction image generation means that generates an instruction image by superimposing a predetermined image on an image captured by the imaging means in the line of sight of the worker in accordance with the operation of an operator who gives instructions to the worker, and the head-mounted display device includes a control means that controls the instruction image generated by the instruction image generation means so that the instruction image overlaps with the field of view of the worker when it is displayed. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent Publication No. 2021-039567 [Patent Document 2] Patent No. 4553362 [Patent Document 3] Japanese Patent Application Laid-Open No. 2013-16020 Summary of the Invention [Problem to be solved by the invention]

[0006] A technology has been proposed in which video footage taken by on-site workers is sent to a terminal operated by a remote support person, allowing the on-site workers and the remote support person to share the work situation and provide support for the work.

[0007] Incidentally, there is a technology that detects the hand movements of a supporter using a camera or the like, and then superimposes an image (hereinafter referred to as an "instruction image") showing the supporter's instructions based on the detected results onto video captured by a worker, thereby displaying the instruction image at the relevant location in the video.

[0008] However, when a worker shoots video while working, the distance between the camera and the work object fluctuates, which can change the size of the work object in the video. Therefore, it is necessary to change the amount of movement and position of the superimposed instruction image each time depending on the change in distance, which does not always allow clear instructions to be conveyed.

[0009] The present invention aims to provide an information processing device, an information processing system, and an information processing program that can convey clearer instructions compared to when the movement amount and position of a superimposed instruction image are changed each time in accordance with changes in the distance between the imaging unit and the work object. [Means for solving the problem]

[0010] An information processing device of a first aspect has a processor, which acquires video of a work object on which work is to be performed and instructions to generate a still image from the video, generates a still image cut out from the video including the work object in accordance with the instructions, uses the still image to identify the work object in the video, position information which is information about the position of the work object, and a superimposition area in which to superimpose an image based on the position of the work object, accepts instruction information which indicates work instructions for the work object, and displays an instruction image which is an image corresponding to the instruction information superimposed on the superimposition area in the video.

[0011] In the information processing device of the second aspect, in the information processing device of the first aspect, the processor further acquires spatial information, which is information of a three-dimensional space corresponding to the video and including the work object, detects feature points indicating the work object from the still image, uses the feature points to identify the work object and position information in the spatial information, and uses the position information to set a superimposition space corresponding to the superimposition area in the spatial information.

[0012] In a third aspect of the information processing device, in the information processing device according to the second aspect, the processor detects a reference point in the work object, and in the spatial information, corresponds the reference point to the center point of the superimposed space to set the superimposed space.

[0013] An information processing device of a fourth aspect is an information processing device relating to any one of the first to third aspects, in which the instruction information is information on detected hand movements, and the processor displays an instruction image corresponding to the movement in an overlapping area in the video.

[0014] An information processing device of a fifth aspect is the information processing device according to the fourth aspect, wherein the processor further acquires a range in which a motion is detected, and sets a superimposition region corresponding to the range.

[0015] In a sixth aspect of the information processing device, in the information processing device according to the fifth aspect, the processor detects the distance to a reference point on the work object, and the greater the distance, the smaller at least one of the size of the image displayed in the superimposition area and the amount of movement of the instruction image in response to the action is reduced.

[0016] An information processing system of a seventh aspect includes an information processing device according to any one of the first to sixth aspects and a terminal that detects instruction information from a user, wherein the information processing device transmits a still image including an overlay area, and the terminal acquires the still image, detects the user's hand movements as instruction information, and displays an instruction image corresponding to the movement superimposed on the overlay area in the still image.

[0017] An information processing program of an eighth aspect causes a computer to acquire video of a work object that is to be worked on and instructions to generate a still image from the video, generate a still image cut out from the video including the work object in accordance with the instructions, use the still image to identify the work object in the video, position information that is information about the position of the work object, and a superimposition area in which to superimpose an image based on the position of the work object, accept instruction information that indicates work instructions for the work object, and display an instruction image that is an image corresponding to the instruction information superimposed on the superimposition area in the video. [Effects of the Invention]

[0018] According to the information processing device of the first aspect and the information processing program of the eighth aspect, clearer instructions can be conveyed compared to when the movement amount and position of the superimposed instruction image are changed each time in accordance with changes in the distance between the imaging unit and the work object.

[0019] According to the information processing device of the second aspect, even if the relative positions of the object and the image capturing unit change, the instruction image can be displayed.

[0020] According to the information processing device of the third aspect, it is possible to display the instruction image in a range fixed relative to the work object.

[0021] According to the information processing device of the fourth aspect, it is possible to show instructions to the video that the worker is looking at.

[0022] According to the information processing device of the fifth aspect, the worker can recognize the detection range that the assistant recognizes.

[0023] According to the information processing device of the sixth aspect, instructions from the supporter can be conveyed more accurately compared to a case where the size of the image to be displayed according to the distance and the amount of movement of the instruction image according to the action do not change.

[0024] According to the information processing system of the seventh aspect, the assistant can give instructions while looking at the same image as the worker is looking at. [Brief explanation of the drawings]

[0025] [Figure 1] 1 is a schematic diagram illustrating an example of a configuration of an information processing system according to an embodiment of the present invention. [Figure 2] FIG. 2 is a block diagram showing an example of a hardware configuration of the information processing device according to the present embodiment. [Figure 3] FIG. 2 is a block diagram illustrating an example of a functional configuration of the information processing device according to the present embodiment. [Figure 4] 10A and 10B are schematic diagrams showing an example of a still image provided for explaining setting of an overlapping region according to the present embodiment. [Figure 5] 10A and 10B are schematic diagrams showing an example of reference points for setting an overlapping region according to the present embodiment; [Figure 6] FIG. 2 is a schematic diagram showing an example of three-dimensional space information according to the present embodiment. [Figure 7] FIG. 4 is a schematic diagram illustrating an example of an overlapping region according to the present embodiment. [Figure 8] FIG. 1 is a schematic diagram showing an example of three-dimensional space information for explaining SLAM according to the present embodiment. [Figure 9] FIG. 2 is a block diagram showing an example of a hardware configuration of a terminal according to the present embodiment. [Figure 10] FIG. 2 is a block diagram illustrating an example of a functional configuration of a terminal according to the present embodiment. [Figure 11] FIG. 4 is a schematic diagram showing an example of a detection range according to the present embodiment. [Figure 12] FIG. 4 is a sequence diagram illustrating an example of information processing according to the present embodiment. [Figure 13] 10 is a flowchart illustrating an example of a process for superimposing an instruction image according to the present embodiment. [Figure 14] 10 is a flowchart illustrating an example of a process for detecting instruction information according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0026] [First embodiment] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will now be described in detail with reference to the accompanying drawings. Fig. 1 is a schematic diagram showing an example of the configuration of an information processing system 1 according to this embodiment.

[0027] 1, an information processing system 1 includes an information processing device 10 operated by a worker and a terminal 50 operated by a supporter. The information processing device 10 and the terminal 50 are connected to each other via a network N.

[0028] The information processing device 10 is a terminal such as a tablet or mobile terminal equipped with a monitor 16 and a camera 18, which will be described later. The information processing device 10 acquires an image including an object to be worked on (hereinafter referred to as "work object") and transmits the acquired image to the terminal 50. The information processing device 10 also acquires information regarding work instructions from the supporter (hereinafter referred to as "instruction information") from the terminal 50, and presents an image corresponding to the instruction information to the worker by superimposing it on the image.

[0029] The terminal 50 acquires the video from the information processing device 10 and presents it to the supporter. The terminal 50 also transmits instruction information input by the supporter to the information processing device 10.

[0030] In the information processing system 1, the information processing device 10 transmits video captured by the worker and presents it on the terminal 50, and the terminal 50 transmits instruction information input by the supporter and presents it on the information processing device 10. The information processing system 1 enables the worker to receive instruction information from a remote supporter via the information processing device 10 and perform work on a work target. Note that in this embodiment, a form in which continuously captured images are acquired as video will be described.

[0031] Next, the hardware configuration of the information processing device 10 will be described with reference to Fig. 2. Fig. 2 is a block diagram showing an example of the hardware configuration of the information processing device 10 according to this embodiment.

[0032] 2, the information processing device 10 according to this embodiment includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage 14, an input unit 15, a monitor 16, a communication interface (communication I / F) 17, and a camera 18. The CPU 11, the ROM 12, the RAM 13, the storage 14, the input unit 15, the monitor 16, the communication I / F 17, and the camera 18 are connected to each other via a bus 19. Here, the CPU 11 is an example of a processor.

[0033] The CPU 11 supervises and controls the entire information processing device 10. The ROM 12 stores various programs, including the information processing program used in this embodiment, and data. The RAM 13 is a memory used as a work area when various programs are executed. The CPU 11 loads the program stored in the ROM 12 into the RAM 13 and executes it, thereby displaying an instruction image on the screen. The storage 14 is, for example, a hard disk drive (HDD), a solid state drive (SSD), or flash memory. The storage 14 may store an information processing program, etc. The input unit 15 is, for example, a touch panel that accepts character input, a keyboard, etc. The monitor 16 displays characters and images. The communication I / F 17 transmits and receives data. The camera 18 is an imaging device that captures an image of a work target. Here, the camera 18 is an example of an "imaging unit."

[0034] Next, the functional configuration of the information processing device 10 will be described with reference to Fig. 3. Fig. 3 is a block diagram showing an example of the functional configuration of the information processing device 10 according to this embodiment.

[0035] 3 , the information processing device 10 includes an acquisition unit 21, a reception unit 22, a generation unit 23, a detection unit 24, a setting unit 25, an estimation unit 26, an identification unit 27, a transmission unit 28, and a display unit 29. The CPU 11 executes an information processing program to function as the acquisition unit 21, the reception unit 22, the generation unit 23, the detection unit 24, the setting unit 25, the estimation unit 26, the identification unit 27, the transmission unit 28, and the display unit 29.

[0036] The acquisition unit 21 acquires an image including an object to be worked on, captured by the camera 18. Note that in this embodiment, the work object is described as being identified by reading a QR (Quick Response) code attached to the surface of the object. However, this is not limiting. The identified object may be displayed on the monitor 16, and the user may select a work object from the displayed objects.

[0037] The receiving unit 22 receives instructions to generate a still image from the video, and information indicating instructions for work on a work target from the terminal 50 (hereinafter referred to as "instruction information"). Here, the instruction information is, for example, information on the supporter's hand movements detected in order for the supporter to indicate instructions with hand movements. In addition to the instruction to generate a still image, the receiving unit 22 also receives information on the range in which the supporter's hand movements (described later) are to be detected (hereinafter referred to as "detection range").

[0038] When the generation unit 23 receives an instruction to generate a still image, the generation unit 23 cuts out an image from the acquired video and generates the still image.

[0039] The detection unit 24 detects feature points indicating an object from the acquired video. The detection unit 24 also uses the generated still image to detect a reference point and the distance to the surface of the work object as position information of the work object. Here, feature points are points that indicate characteristics such as edges and corners of an object included in the video. Furthermore, as shown in FIG. 4 as an example, the reference point 30 is a point that serves as a reference for the position of the work object 32, where the center of the still image 31 and the work object 32 included in the still image 31 overlap. The distance to the work object 32 is the distance between the camera 18 and the reference point 30 shown in FIG. 4. In other words, as shown in FIG. 5 as an example, the detection unit 24 detects the distance from the camera 18 to the reference point 30, where the collimation axis 33 of the camera 18, which corresponds to the center of the still image 31, intersects with the work object 32.

[0040] The setting unit 25 sets three-dimensional space information using the detected feature points. Specifically, the setting unit 25 sets the three-dimensional space information shown in Fig. 6 as an example. As shown in Fig. 6, the setting unit 25 uses feature points 34 to identify the space included in the video and each object including the work target 32.

[0041] Furthermore, the setting unit 25 sets, in the three-dimensional space information, a space (hereinafter referred to as a "superimposed space") for superimposing an image corresponding to the instruction information (hereinafter referred to as an "instruction image") using the distance to the work object 32 detected by the detection unit 24. For example, the setting unit 25 sets, in the three-dimensional space information, a superimposed space of a size corresponding to the size of the detection range using the distance to the work object 32. Specifically, if the detection range for detecting the supporter's hand movements is 50 cm wide, 50 cm high, and 50 cm deep, the setting unit 25 similarly sets, in the three-dimensional space information, a space of 50 cm wide, 50 cm high, and 50 cm deep as the superimposed space. Furthermore, as an example, as shown in FIG. 7, the setting unit 25 sets the superimposed space so that the center of the superimposed space 35 corresponds to the reference point 30 in the three-dimensional space.

[0042] That is, the setting unit 25 sets, in the three-dimensional space information, a superimposed space 35 having a size according to the detection range with respect to the reference point 30 as the base for the work object 32. By setting the superimposed space 35 having a size according to the detection range, the ratio of the superimposed space 35 to the work object 32 is determined to be constant, and the size of the space recognized by the supporter and the size of the space recognized by the worker through the monitor 16 are associated with each other.

[0043] The estimation unit 26 estimates the position and direction of the camera 18 (worker) using the detected feature points 34, and estimates an overlap space 35 at the position and direction of the camera 18 (worker). Specifically, the estimation unit 26 estimates position information indicating the position of the camera 18 (worker), direction information indicating the direction, and the overlap space 35 using SLAM (Simultaneous Localization And Mapping) technology.

[0044] For example, a worker starts capturing an image by reading a QR code (registered trademark) attached to the work object 32, and the detection unit 24 detects feature points 34 included in the captured image. The estimation unit 26 compares the feature points 34 included in multiple images captured over time and estimates the position and direction of the camera 18 (worker) from the amount of change in the feature points 34. The estimation unit 26 uses the estimated position information and direction information to estimate the position of the superimposed space 35 in the three-dimensional space information. By estimating the position of the superimposed space 35, an instruction image is displayed in the captured image even if the worker's position changes.

[0045] As an example, as shown in FIG. 8 , by tracking feature points 34 included in a captured image, the worker's position information, directional information, and superimposed space 35 can be estimated. In the present embodiment, the worker's position at the start of image capture is estimated by reading a QR code (registered trademark) attached to the work object 32. However, this is not limited to this. For example, image capture of the work object 32 may be started at a predetermined position. In the present embodiment, the worker's position information and directional information are estimated from the amount of change in the feature points 34. However, this is not limited to this. For example, a feature point map in which feature points 34 of the space where the worker is located is created in advance, and the worker's position information and directional information may be estimated by comparing the feature points 34 included in the captured image with the feature point map.

[0046] The identification unit 27 uses the three-dimensional space information to identify a superimposition area in the video and still image 31 that corresponds to the superimposition space 35. Specifically, the identification unit 27 compares the video and still image 31 with the three-dimensional space information to identify the work object 32 and position information of the work object 32 in the video and still image 31. Furthermore, the identification unit 27 uses the position information of the identified work object 32 to identify a superimposition area in the video and still image 31 that corresponds to the superimposition space 35.

[0047] The transmitter 28 transmits the video and the still image 31 to the terminal 50. Here, the transmitter 28 transmits the distance to the work object 32 in the still image 31 together with the still image 31.

[0048] The display unit 29 displays an instruction image corresponding to the received instruction information by superimposing it in the superimposition area on the video and still image 31. The display unit 29 also switches the video and still image displayed on the monitor 16 in response to an instruction from the worker. Here, the display unit 29 displays an instruction image of a size corresponding to the distance to the work object 32. For example, the greater the distance to the work object 32, the smaller the instruction image to be displayed.

[0049] In the present embodiment, the explanation has been given of a mode in which the greater the distance to the work object 32, the smaller the displayed instruction image. However, this is not limiting. The amount of movement of the instruction image may be changed depending on the distance to the work object 32. For example, the display unit 29 may move and display the instruction image in accordance with the action in the instruction information, and the greater the distance to the work object 32, the smaller the amount of movement of the instruction image.

[0050] Next, the hardware configuration of the terminal 50 will be described with reference to Fig. 9. Fig. 9 is a block diagram showing an example of the hardware configuration of the terminal 50 according to this embodiment.

[0051] 9, the terminal 50 according to this embodiment includes a CPU 51, a ROM 52, a RAM 53, a storage 54, an input unit 55, a monitor 56, a communication interface (communication I / F) 57, and a detection device 58. The CPU 51, the ROM 52, the RAM 53, the storage 54, the input unit 55, the monitor 56, the communication I / F 57, and the detection device 58 are connected to each other via a bus 59.

[0052] The CPU 51 supervises and controls the entire terminal 50. The ROM 52 stores various programs, data, etc., including a detection processing program used in this embodiment. The RAM 53 is a memory used as a work area when various programs are executed. The CPU 51 performs processing to detect instruction information by expanding the programs stored in the ROM 52 into the RAM 53 and executing them. The storage 54 is, for example, an HDD, SSD, or flash memory. The storage 54 may also store the detection processing program, etc. The input unit 55 is, for example, a touch panel and a keyboard that accepts character input, etc. The monitor 56 displays characters and images. The communication I / F 57 transmits and receives data. The detection device 58 is, for example, a camera that detects the hand movements of the supporter. In this embodiment, the detection device 58 is a camera. However, this is not limiting. The detection device 58 may also be a sensor.

[0053] Next, the functional configuration of the terminal 50 will be described with reference to Fig. 10. Fig. 10 is a block diagram showing an example of the functional configuration of the terminal 50 according to this embodiment.

[0054] 10 , the terminal 50 includes a reception unit 61, an image acquisition unit 62, a motion detection unit 63, a setting unit 64, a display unit 65, and a transmission unit 66. The CPU 51 executes a detection processing program, thereby functioning as the reception unit 61, the image acquisition unit 62, the motion detection unit 63, the setting unit 64, the display unit 65, and the transmission unit 66.

[0055] The receiving unit 61 receives an instruction to generate the still image 31 from the supporter.

[0056] The image acquisition unit 62 acquires video and still images 31 from the information processing device 10. Here, the image acquisition unit 62 acquires the still images 31 from the information processing device 10 as well as the distance to the work object 32 in the still images 31.

[0057] The movement detection unit 63 detects the supporter's hand movements as instruction information using the detection device 58. Here, the movement detection unit 63 detects the shape and three-dimensional position of the supporter's hand by analyzing multiple images including the supporter's hand taken by the detection device 58, and detects the hand movements as instruction information.

[0058] The setting unit 64 sets a superimposition area corresponding to the detection range for detecting motion in the acquired still image 31. Specifically, the setting unit 64 sets a superimposition area corresponding to the detection range for the work object 32 included in the still image 31 using the acquired distance to the work object 32. For example, if the detection range is 50 cm wide, 50 cm high, and 50 cm deep, the setting unit 64 estimates the size of the work object 32 using the acquired distance to the work object 32 and sets a superimposition area corresponding to the detection range in the still image 31. Note that the superimposition area on the still image 31 is set so that a reference point on the still image 31 corresponds to the center point of the superimposition area. Here, the reference point is the position where the center of the still image 31 and the work object 32 included in the still image 31 overlap.

[0059] The display unit 65 displays an instruction image corresponding to the detected instruction information in the superimposed area on the still image 31.

[0060] 11 as an example, the movement detection unit 63 detects, as instruction information, the hand movement of the supporter that is included in the detection range 70 set by the detection device 58. When detecting the hand movement, the display unit 65 displays on the monitor 56 the acquired still image 31 and an instruction image corresponding to the instruction information in a superimposed area on the still image 31. By displaying the still image 31 with the instruction image superimposed in the superimposed area by the display unit 65, the supporter can give clear instructions while checking the instructions for the work target 32.

[0061] The transmitting unit 66 transmits the instruction information to the information processing device 10. The transmitting unit 66 also transmits an instruction to generate the still image 31 to the information processing device 10.

[0062] Next, the operation of the information processing system 1 in which the information processing device 10 and the terminal 50 cooperate will be described with reference to Fig. 12. Fig. 12 is a sequence diagram showing an example of the flow of the information processing system of this embodiment.

[0063] As an example, as shown in FIG. 12, the information processing device 10 acquires an image captured by the camera 18 (step S101), and transmits the acquired image to the terminal 50 (step S102).

[0064] The terminal 50 acquires the video from the information processing device 10 and displays it on the monitor 56 (step S103). When the terminal 50 receives an instruction to generate the still image 31 from the supporter, the terminal 50 transmits the instruction to generate the still image 31 to the information processing device 10 (step S104). Here, the terminal 50 transmits the size of the detection range together with the instruction to generate the still image 31.

[0065] The information processing device 10 receives an instruction to generate a still image 31 from the terminal 50 (step S105), generates the still image 31 by cutting it out from the video (step S106), and transmits the generated still image 31 to the terminal 50 (step S107). Here, the information processing device 10 receives the size of the detection range along with the instruction to generate the still image 31. The information processing device 10 also transmits the distance to the work object 32 along with the generated still image 31.

[0066] The terminal 50 acquires the still image 31 from the information processing device 10 (step S108) and displays the acquired still image 31 on the monitor 56 (step S109). The terminal 50 detects instruction information, which is the supporter's hand movement (step S110), and displays an instruction image corresponding to the instruction information by superimposing it on the still image 31 (step S111). The terminal 50 transmits the detected instruction information to the information processing device 10 (step S112). Here, the terminal 50 acquires the distance to the work object 32, sets a superimposition area on the still image 31 using the distance to the work object 32, and displays an instruction image corresponding to the instruction information in the superimposition area.

[0067] The information processing device 10 acquires instruction information from the terminal 50 (step S113), and displays an instruction image according to the instruction information superimposed on the superimposition area set on the video (step S114).

[0068] Next, the operation of the information processing device 10 according to this embodiment will be described with reference to Fig. 13. Fig. 13 is a flowchart showing an example of processing for displaying an instruction image according to instruction information according to this embodiment. The information processing shown in Fig. 13 is executed by the CPU 11 reading and executing an information processing program from the ROM 12 or the storage 14. The information processing shown in Fig. 13 is executed, for example, when an instruction to display an instruction image is input by the user.

[0069] In step S201, the CPU 11 acquires a video of an object including the work target 32.

[0070] In step S202, the CPU 11 identifies the work object 32 included in the video.

[0071] In step S203, the CPU 11 detects feature points 34 of the object from the acquired image.

[0072] In step S204, the CPU 11 sets the detected feature points in the three-dimensional space information.

[0073] In step S205, the CPU 11 estimates the position information and direction information of the worker from the detected feature points 34 using the SLAM technique.

[0074] In step S206, the CPU 11 displays the acquired video on the monitor 16.

[0075] In step S207, the CPU 11 determines whether or not an instruction image is set in the three-dimensional space information. If an instruction image is set in the three-dimensional space information (step S207: YES), the CPU 11 proceeds to step S208. On the other hand, if an instruction image is not set in the three-dimensional space information (step S207: NO), the CPU 11 proceeds to step S209.

[0076] In step S208, the CPU 11 displays the instruction image set in the three-dimensional space information superimposed on the video.

[0077] In step S209, the CPU 11 transmits the acquired video to the terminal 50.

[0078] In step S210, the CPU 11 determines whether or not an instruction to generate a still image 31 has been received from the terminal 50. If an instruction to generate a still image 31 has been received from the terminal 50 (step S210: YES), the CPU 11 proceeds to step S211. On the other hand, if an instruction to generate a still image 31 has not been received from the terminal 50 (step S210: NO), the CPU 11 proceeds to step S201 and acquires video.

[0079] In step S211, the CPU 11 acquires the detection range received from the terminal 50 together with an instruction to generate the still image 31.

[0080] In step S212, the CPU 11 generates a still image 31 cut out from the video.

[0081] In step S213, the CPU 11 detects the distance from the still image 31 to the work object 32.

[0082] In step S214, the CPU 11 transmits the generated still image 31 to the terminal 50. Here, the CPU 11 transmits the distance to the work object 32 together with the generated still image 31.

[0083] In step S215, the CPU 11 sets an overlapping space in the three-dimensional space information using the detected distance to the work object 32, and identifies the overlapping area in the video and still image 31.

[0084] In step S216, the CPU 11 determines whether or not instruction information has been received from the terminal 50. If instruction information has been received from the terminal 50 (step S216: YES), the CPU 11 proceeds to step S217. On the other hand, if instruction information has not been received from the terminal 50 (step S216: NO), the CPU 11 waits until instruction information is received from the terminal 50.

[0085] In step S217, the CPU 11 acquires the instruction information received from the terminal 50.

[0086] In step S218, the CPU 11 sets an instruction image corresponding to the acquired instruction information in the superimposed space in the three-dimensional space information.

[0087] In step S219, the CPU 11 uses the three-dimensional space information to superimpose and display the instruction image on the superimposition area in the video.

[0088] In step S220, the CPU 11 determines whether or not to end the process. If the process is to be ended (step S220: YES), the CPU 11 ends the information processing. On the other hand, if the process is not to be ended (step S220: NO), the CPU 11 proceeds to step S201 and acquires the video.

[0089] Next, the operation of the terminal 50 according to this embodiment will be described with reference to Fig. 14. Fig. 14 is a flowchart showing an example of a process for detecting instruction information according to this embodiment. The CPU 51 reads out and executes a detection process program from the ROM 52 or the storage 54, thereby executing the detection process shown in Fig. 14. The detection process shown in Fig. 14 is executed, for example, when an instruction to start assistance is input by the user.

[0090] In step S301, the CPU 51 acquires from the information processing device 10 a video of an object including the work target 32.

[0091] In step S302, the CPU 51 displays the acquired video on the monitor 56.

[0092] In step S303, the CPU 51 determines whether or not it has received an instruction to generate a still image 31. If it has received an instruction to generate a still image 31 (step S303: YES), the CPU 51 proceeds to step S304. On the other hand, if it has not received an instruction to generate a still image 31 (step S303: NO), the CPU 51 proceeds to step S301 and acquires video.

[0093] In step S304, the CPU 51 transmits an instruction to generate the still image 31 to the information processing device 10.

[0094] In step S305, the CPU 51 acquires the still image 31 from the information processing device 10.

[0095] In step S306, the CPU 51 displays the acquired still image 31 on the monitor 56.

[0096] In step S307, the CPU 51 detects instruction information, which is the hand movement of the supporter.

[0097] In step S308, the CPU 51 displays an instruction image corresponding to the detected instruction information superimposed on the still image 31.

[0098] In step S309, the CPU 51 determines whether or not to transmit instruction information to the information processing device 10. If the instruction information is to be transmitted (step S309: YES), the CPU 51 proceeds to step S310. On the other hand, if the instruction information is not to be transmitted (step S309: NO), the CPU 51 proceeds to step S307 and detects the instruction information.

[0099] In step S310, the CPU 51 transmits the detected instruction information to the information processing device 10.

[0100] In step S311, the CPU 51 determines whether or not to end the process. If the process is to be ended (step S311: YES), the CPU 51 ends the detection process. On the other hand, if the process is not to be ended (step S311: NO), the CPU 51 proceeds to step S301 and acquires the video.

[0101] As described above, according to this embodiment, it is possible to convey clearer instructions compared to when the movement amount and position of the superimposed instruction image are changed each time in accordance with changes in the distance between the imaging unit and the work object.

[0102] In the above embodiment, the information processing device 10 is described as a terminal carried by the worker. However, the present invention is not limited to this. The information processing device 10 may be a head-mounted display worn by the worker, or may be a server. For example, a server equipped with the information processing device 10 may acquire an image and a designation of the work target 32 ​​from a terminal carried by the worker, acquire instructions and instruction information for generating a still image 31 from a terminal carried by a supporter, and transmit the image with the instruction image superimposed to the worker's terminal.

[0103] Furthermore, in the present embodiment, a configuration has been described in which the distance to the work object 32 is detected using the still image 31. However, this is not limiting. The distance to the work object 32 may also be detected using a TOF (Time Of Flight) method. For example, when the information processing device 10 receives an instruction to generate the still image 31, it irradiates light from a light source (not shown) and detects the light reflected by the work object 32. The information processing device 10 may detect the distance to the work object 32 by measuring the time from when the light is irradiated to when the reflected light reflected by the work object 32 returns.

[0104] In the present embodiment, a configuration has been described in which a supporter issues an instruction to generate a still image, and the information processing device 10 receives the instruction to generate a still image from the terminal 50. However, the present invention is not limited to this. An operator may issue an instruction to generate a still image.

[0105] Although the present invention has been described above using the embodiments, the present invention is not limited to the scope of the embodiments. Various modifications and improvements can be made to the embodiments without departing from the spirit of the present invention, and such modifications and improvements are also included in the technical scope of the present invention.

[0106] In the above embodiment, the term "processor" refers to a processor in a broad sense, and includes, for example, a general-purpose processor (e.g., a CPU: Central Processing Unit) and a dedicated processor (e.g., a GPU: Graphics Processing Unit, an ASIC: Application Specific Integrated Circuit, an FPGA: Field Programmable Gate Array, a programmable logic device, etc.).

[0107] Furthermore, the operations of the processor in each of the above embodiments may not only be performed by a single processor, but may also be performed by multiple processors located at physically separate locations working together. Furthermore, the order of the operations of the processor is not limited to the order described in each of the above embodiments, and may be changed as appropriate.

[0108] Although the present embodiment has been described with reference to a configuration in which the information processing program is installed in storage, the present invention is not limited to this. The information processing program according to the present embodiment may be provided in a format recorded on a computer-readable storage medium. For example, the information processing program according to the present invention may be provided in a format recorded on an optical disc such as a CD (Compact Disc)-ROM or a DVD (Digital Versatile Disc)-ROM. The information processing program according to the present embodiment may be provided in a format recorded on a semiconductor memory such as a USB (Universal Serial Bus) memory or a memory card. The information processing program according to the present embodiment may also be acquired from an external device via a communication line connected to the communication I / F. [Explanation of symbols]

[0109] 10. Information processing equipment 11, 51 CPUs 12, 52 ROM 13,53 RAM 14, 54 Storage 15, 55 Input section 16, 56 monitors 17, 57 Communication I / F 18 Camera Buses 19 and 59 21 Acquisition Department 22 Reception 23 Generation part 24 Detector 25 Setting section 26 Estimation part 27 Specific section 28 Transmitter 29 Display section 30 reference points 31 still images 32 Work Object 33 Collimation axis 34 minutiae 35 Superimposed Space 50 devices 58 Detection Device 61 Reception Department 62 Image acquisition unit 63 Motion detection unit 64 Settings 65 Display section 66 Transmitter 70 detection range

Claims

1. an information processing device having a processor; a terminal that communicates with the information processing device, The processor: Acquire a video of a work object that is the object of work and an instruction to generate a still image from the video; generating a still image cut out from the video including the work object in response to the instruction; Using the still image, the work object in the video, position information that is information about the position of the work object, and a superimposition area in which an image based on the position of the work object is superimposed are identified; receiving instruction information indicating an instruction for work on the work object from the terminal; superimposing an instruction image, which is an image corresponding to the instruction information, on the superimposed area of ​​the video, and displaying the instruction image on a display unit connected to the terminal and the information processing device; The processor sets a superimposition space having a size corresponding to the size of a detection range of the motion related to the instruction acquired from the terminal. Information processing system.

2. The processor: Further acquiring spatial information corresponding to the image and being information on a three-dimensional space including the work object; Detecting feature points indicative of the work object from the still image; Using the feature points, the work object and the position information in the spatial information are identified; Using the position information, a superimposed space corresponding to the superimposed area is set in the spatial information. The information processing system according to claim 1 .

3. The processor: Detecting a reference point on the work object; In the spatial information, the reference point is associated with the center point of the superimposed space to set the superimposed space. The information processing system according to claim 2 .

4. the instruction information is information about a detected hand movement, The processor: The instruction image corresponding to the action is displayed in the superimposed area of ​​the video. The information processing system according to any one of claims 1 to 3.

5. The processor further obtains a range for detecting the motion; The overlapping area is set corresponding to the range. The information processing system according to claim 4 .

6. The processor: Detecting a distance to a reference point on the work object; The larger the distance, the smaller at least one of the size of the image displayed in the superimposed area and the amount of movement of the instruction image in response to the action. The information processing system according to claim 5 .

7. The information processing device transmits the still image including the superimposed area, The terminal acquiring the still image; detecting a user's hand movement as the instruction information; The instruction image corresponding to the action is superimposed and displayed on the superimposed area of ​​the still image.

7. The information processing system according to claim 1.

8. On the computer, Acquire a video of a work object that is the object of work and an instruction to generate a still image from the video; generating a still image cut out from the video including the work object in response to the instruction; Using the still image, the work object in the video, position information that is information about the position of the work object, and a superimposition area in which an image based on the position of the work object is superimposed are identified; receiving instruction information indicating an instruction for work on the work object from a terminal; an instruction image corresponding to the instruction information is superimposed on the superimposed area of ​​the video; A superimposition space having a size corresponding to the size of a detection range of the motion related to the instruction acquired from the terminal is set. An information processing program that executes the following:

Images