Information Processing Apparatus, Information Processing Program, and Information Processing System

Abstract

To provide an information processing apparatus configured to present a fingertip object while changing a representation of the fingertip object according to information input by a user when superimposing the fingertip object representing an instruction of a user on a video of a target, an information processing program, and an information processing system.SOLUTION: In an information processing system comprising an engineer-side device 1 and an expert-side device 20, an information processing apparatus is mounted on each of the engineer-side device and the expert-side device 20, and comprises a processor. The processor is configured to: acquire a target video showing a target 40 of an instruction, and fingertip information indicating an actual motion of a hand 42 of a user; superimpose a fingertip object 41 indicated by the fingertip information on the target video; and present the fingertip object 41 while changing a representation of the fingertip object according to, input information included in the fingertip information or input by the user.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing program, and an information processing system.

Background Art

[0002] Non-Patent Document 1 discloses a technique in which when the pose of a virtual camera that renders 3D content is adjusted to the pose of the camera of a device provided by ARCore, a developer can render virtual content from a correct viewpoint, and the rendered virtual image is overlaid on the image acquired from the device's camera to make the virtual content appear as if it were part of the real world.

Prior Art Documents

Patent Documents

[0003]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When giving instructions to a user at a remote location regarding a specific object at the site, communication is carried out using a video showing the object. At this time, it is difficult to accurately convey the content of the instructions only by oral instructions using the video showing the object. Even when communicating by overlapping the movement of the user's hand on the video showing the object, the object indicated by the movement of the user's hand may be misunderstood.

[0005] An object of the present invention is to provide an information processing apparatus and an information processing program capable of changing and presenting the expression by a fingertip object according to information input by a user when a fingertip object representing a user's instruction is superimposed and displayed on an image of an object.

Means for Solving the Problems

[0006] The information processing apparatus according to the first aspect includes a processor. The processor acquires an object image showing an object of an instruction and fingertip information representing the movement of a user's hand, superimposes and displays a fingertip object represented by the fingertip information on the object image, and changes and presents the expression by the fingertip object according to the input information included in the fingertip information or input from the user.

[0007] In the information processing apparatus according to the second aspect, when the object indicated by the fingertip object is the object, the processor presents the fingertip object in a predetermined manner according to the input information.

[0008] In the information processing apparatus according to the third aspect, the input information is the movement of the user's hand, and the processor presents the expression indicated by the fingertip object by distinguishing according to the mode of the movement of the user's hand.

[0009] In the information processing apparatus according to the fourth aspect, the mode of the movement of the user's hand is a mode of moving the user's fingertip up and down. When the object indicated by the fingertip object is the object according to the number of times the user's fingertip moves up and down, the processor presents the fingertip object in a predetermined manner.

[0010] In the information processing apparatus according to the fifth aspect, the mode of the movement of the user's hand is a mode of rotating the user's fingertip. When the object indicated by the fingertip object is the object according to the direction of rotation of the user's fingertip, the processor presents the fingertip object in a predetermined manner.

[0011] In the information processing apparatus according to the sixth aspect, the mode of movement of the user's hand is an operation of contacting the output unit on which the object image is displayed, and when the processor contacts the output unit, the fingertip object is presented in a predetermined mode.

[0012] In the information processing apparatus according to the seventh aspect, the object indicated by the mode of movement of the user's hand using a predetermined finger is the object, and when the object indicated by the fingertip object is the object, the processor presents the fingertip object in a predetermined mode.

[0013] In the information processing apparatus according to the eighth aspect, the processor presents the fingertip object superimposed on the object image in a distinguishable manner according to the acquired input information.

[0014] In the information processing apparatus according to the ninth aspect, when the object indicated by the fingertip object is the object, the processor presents an indication line extending from the tip of the fingertip of the fingertip object superimposed on the object image toward the object.

[0015] In the information processing apparatus according to the tenth aspect, the processor changes the presentation mode of the indication line according to the input information.

[0016] In the information processing apparatus according to the eleventh aspect, when the object indicated by the fingertip object is the object, the processor presents the fingertip object in a distinguishable manner by the color of the fingertip object.

[0017] In the information processing apparatus according to the twelfth aspect, when the object indicated by the fingertip object is the object, the processor presents the fingertip object in a distinguishable manner by sound.

[0018] In the information processing apparatus according to the 13th aspect, the processor returns the presentation of the fingertip object to its original state and presents it based on a change in the presentation mode of the fingertip object due to an instruction by the user or the elapse of a predetermined time.

[0019] In the information processing apparatus according to the 14th aspect, the instruction by the user is an operation of moving the user's hand to a specific area, and when the processor detects the movement of the user's hand to the specific area, the processor returns the change in the mode of the fingertip object to its original state and presents it.

[0020] In the information processing apparatus according to the 15th aspect, when the contact with the output unit is released, the processor returns the change in the mode of the fingertip object to its original state and presents it.

[0021] In the information processing program according to the 16th aspect, a computer acquires an object image reflecting an object to be instructed and fingertip information representing the movement of the user's hand, overlays and displays a fingertip object represented by the fingertip information on the object image, and changes and presents the expression by the fingertip object according to the fingertip information included therein or input information input from the user.

[0022] The information processing system according to the 17th aspect is an information processing system including a first device and a second device. The first device includes a first processor, the second device includes a second processor, the first processor acquires an object image reflecting an object to be instructed and displays the object image on a display unit, the second processor acquires fingertip information representing the movement of the user's hand, transmits the fingertip information to the first device so as to overlay and display a fingertip object represented by the fingertip information on the object image on the display unit, and the first processor changes and presents the expression by the fingertip object according to the fingertip information included therein or input information input from the user on the display unit.

Advantages of the Invention

[0023] According to the information processing apparatus according to the first aspect, when a fingertip object representing a user's instruction is superimposed on a video of an object and displayed, the expression by the fingertip object can be changed and presented according to the information input by the user.

[0024] According to the information processing apparatus according to the second aspect, when the object pointed by the fingertip object is the object according to the input information, the fingertip object can be presented in a predetermined manner.

[0025] According to the information processing apparatus according to the third aspect, the expressions indicated by the fingertip object can be presented separately according to the manner of movement of the user's hand.

[0026] According to the information processing apparatus according to the fourth aspect, when the object pointed by the fingertip object is the object according to the number of times the user moves the fingertip up and down, the fingertip object can be presented in a predetermined manner.

[0027] According to the information processing apparatus according to the fifth aspect, when the object pointed by the fingertip object is the object according to the direction of rotation of the user's fingertip, the fingertip object can be presented in a predetermined manner.

[0028] According to the information processing apparatus according to the sixth aspect, when contacting the output unit, the fingertip object can be presented in a predetermined manner.

[0029] According to the information processing apparatus according to the seventh aspect, when the object indicated by the manner of movement of the user's hand using a predetermined finger is the object and the object pointed by the fingertip object is the object, the fingertip object can be presented in a predetermined manner.

[0030] According to the information processing apparatus according to the eighth aspect, according to the acquired input information, the expression modes of the tip portions of the fingertips of the fingertip objects superimposed and displayed on the object video can be presented separately.

[0031] According to the information processing apparatus according to the ninth aspect, when the object indicated by the fingertip object is an object, an instruction line can be presented from the tip of the fingertip of the fingertip object displayed overlaid on the object video, toward the object.

[0032] According to the information processing apparatus according to the tenth aspect, the presentation mode of the instruction line can be changed according to the input information.

[0033] According to the information processing apparatus according to the eleventh aspect, when the object indicated by the fingertip object is an object, it can be presented by distinguishing according to the color of the fingertip object.

[0034] According to the information processing apparatus according to the twelfth aspect, when the object indicated by the fingertip object is an object, it can be presented by distinguishing by sound.

[0035] According to the information processing apparatus according to the thirteenth aspect, based on the change in the presentation mode of the fingertip object, it can be presented after restoring it, according to an instruction by the user or the elapse of a predetermined time.

[0036] According to the information processing apparatus according to the fourteenth aspect, when it is detected that the user's hand has moved to the specific area, based on the change in the mode of the fingertip object, it can be presented after restoring it.

[0037] According to the information processing apparatus according to the fifteenth aspect, when the contact with the output unit is released, based on the change in the mode of the fingertip object, it can be presented after restoring it.

[0038] According to the information processing program according to the sixteenth aspect, when displaying a fingertip object representing a user's instruction overlaid on a video showing an object, according to the information input by the user, the expression by the fingertip object can be changed and presented.

[0039] According to the information processing system according to the 17th aspect, when a fingertip object representing a user's instruction is superimposed on a video of an object and displayed, the expression by the fingertip object can be changed and presented according to the information input by the user.

Brief Description of the Drawings

[0040]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Embodiments for Carrying Out the Invention

[0041] Hereinafter, an example of an embodiment according to the disclosed technology will be described in detail with reference to the drawings.

[0042] FIG. 1 is a diagram showing a schematic configuration of an information processing system including an information processing apparatus 1 according to the present embodiment.

[0043] The information processing apparatus 1 according to this embodiment is, for example, a device used when an expert gives an instruction regarding a specific object to an engineer located at a remote location. The information processing apparatus 1 causes the object to be displayed on each of the devices on the expert side and the engineer side, and overlays and displays an object indicating the movement of the expert's hand on the displayed video of the object. The information processing apparatus 1 discriminates and presents the expression by the object according to the information input by the expert. Hereinafter, the object indicating the movement of the expert's hand, which is overlaid on the video of the object, is expressed as a fingertip object.

[0044] The information processing apparatus 1 according to this embodiment is mounted on each of the engineer-side device 10 and the expert-side device 20. As shown in FIG. 1, the engineer-side device 10 and the expert-side device 20 are communicably connected. Although not shown, these devices are connected to a network and can communicate with each other via the network. As an example, the Internet, a LAN (Local Area Network), a WAN (Wide Area Network), etc. are applied to this network. Further, the engineer-side device 10 and the expert-side device 20 may be connected to an input device, an output device, an imaging device, etc. as external devices by these networks.

[0045] In FIG. 1, the engineer-side device 10 is represented as a smartphone, but it is not limited to tablet-type terminals. Also, the expert-side device 20 is represented as a general computer terminal, but it may be a tablet-type terminal such as a smartphone and is not limited to general computer terminals. Both are equipped with a display or the like as an output device for displaying video or images. The display as the output device may also have a function as an input device such as a touch panel. Also, the output device and the input device do not necessarily have to be mounted inside the engineer-side device 10 and the expert-side device 20, and may be connected as external devices to each. Furthermore, the output such as screen display is not limited to a display or the like, and may be VR (Virtual Reality) glasses or the like.

[0046] As shown in FIG. 1, a sensor 30 is connected to the expert-side device 20. The sensor 30 is a motion sensor that captures the actual human hand to recognize the position and shape of each joint and finger, and can recognize the movement of the finger with high precision. It is a device for sensing the actual hand movement of the expert and acquiring the hand movement as information. As an example, the leap motion (registered trademark) controller of Ultraleap is used for the sensor 30. The leap motion controller is an optical hand tracking module that captures the movement of a human hand with high precision, recognizes the position of the fingers and joints, and precisely senses the movement of each finger. The motion sensor used as the sensor 30 may be any sensor as long as it can sense the movement of the user's finger, and is not limited to the above-mentioned leap motion (registered trademark) controller.

[0047] FIG. 2 is a diagram showing an example of the display output to the expert-side device 20. The expert-side device 20 displays the video of the object 40 to be instructed, which is transmitted from the engineer-side device 10. Hereinafter, the video of the object 40 is referred to as the object video. While displaying the object video transmitted from the engineer-side device 10 on the expert-side device 20, the expert uses the fingertip object 41 superimposed and displayed on the object video to operate the sensor 30 to give instructions regarding the object 40 to the engineer located at a remote location.

[0048] As shown in FIG. 2, the expert uses the sensor 30 connected to the expert-side device 20 to sense the actual hand movement of the expert. The hand 42 shown in FIG. 2 is the actual hand of the expert, and the hand object 41 displayed on the expert-side device 20 is the fingertip object 41 displayed on the screen.

[0049] The fingertip object 41 is an object displayed on the screen to reflect the actual hand movement of the expert. The display of the fingertip object 41 is not limited to the deformed hand object as an illustration as shown in FIG. 2, and an object in the shape of an arrow or the actual hand of the captured expert may be used as it is.

[0050] In FIG. 2, as an example, the object 40 is represented by an object in which five cylindrical shapes are connected in a row. In this case, with only the object video and oral instructions, when giving instructions, the expert has to verbally express "the third cylinder from the left", "the frontmost cylinder", or "the third cylinder from the right" one by one, making the instructions cumbersome. Also, even if it is expressed verbally as "around the middle of that cylinder", it is difficult to understand the actual target location on the object 40, and there is a risk of misunderstanding in the communication between each other. Therefore, in the information processing apparatus 1 according to the present embodiment, by using the fingertip object 41 superimposed and displayed on the object video, it is possible to accurately give instructions regarding the object 40 to the engineer located at a remote location.

[0051] Furthermore, the information processing apparatus 1 according to the present embodiment changes the expression by the fingertip object 41 according to the information input by the expert depending on the object indicated by the fingertip object 41, and presents it in a distinguishable manner before and after the expert inputs the information. For example, as the object indicated by the fingertip object 41, there may be a case where it indicates the object 40, a case where it indicates the direction of the display screen, or a case where it is simply displayed on the display screen. Therefore, if it is not clear what the object indicated by the fingertip object 41 is, there is a risk of misunderstanding in mutual communication. Thus, the information processing apparatus 1 according to the present embodiment presents the mode of the fingertip object 41 in a distinguishable manner depending on the object indicated by the fingertip object 41.

[0052] FIG. 3 is a diagram showing an example of the display on the engineer side device 10. The engineer side device 10 captures the object 40 that requires an instruction from the expert, and transmits the captured object video to the expert side device 20. The basic operation of the expert side device 20 that has received the object video is as shown in FIG. 2. By the engineer side device 10 receiving the movement of the expert's hand acquired by the expert side device 20, as shown in FIG. 3, the fingertip object 41 is superimposed and displayed on the object video in the engineer side device 10. The fingertip object 41 displayed on the engineer side device 10 is presented in a mode that clarifies the object indicated by the fingertip object 41.

[0053] Next, the hardware configuration of the engineer side device 10 according to the present embodiment will be described. FIG. 4 is a block diagram showing the hardware configuration of the engineer side device 10 according to the present embodiment.

[0054] As shown in FIG. 4, the engineer side device 10 includes a CPU (Central Processing Unit) 11, a memory 12, a storage unit 13, a communication I / F (Interface) 14, an input / output I / F 15, an input unit 16, an output unit 17, a storage medium reader 18, and an imaging unit 19.

[0055] The CPU 11 is a central processing unit that executes various programs and controls each component. That is, the CPU 11 reads a program from the storage unit 13 and executes the program using the memory 12 as a work area. The CPU 11 performs control of each of the above components and various arithmetic processes according to the program stored in the storage unit 13.

[0056] The memory 12 is composed of a RAM (Random Access Memory) and temporarily stores programs and data as a work area. The storage unit 13 is composed of a ROM (Read Only Memory), an HDD (Hard Disk Drive), an SSD (Solid State Drive), or the like, and stores various programs including an operating system and various data.

[0057] The communication I / F 14 is an interface for communicating with other devices, and for example, standards such as Ethernet (registered trademark), FDDI, or Wi-Fi (registered trademark) are used.

[0058] The input / output I / F 15 is an interface that connects the engineer-side device 10 and an external device. In the present embodiment, the engineer-side device 10 is connected to an external input device or the like via the input / output I / F 15.

[0059] The input unit 16 is a configuration for performing various inputs, such as a keyboard and a mouse. The input unit 16 may perform input by voice. When connecting an imaging device as an external device, input of video and images captured using the imaging device is also performed.

[0060] The output unit 17 is, for example, a display or the like, and is a configuration for outputting a display of an object video, a fingertip object 41, and the like. The output by the output unit 17 is not limited to video, and may include output of sound or voice. In that case, the output unit 17 includes a speaker or the like.

[0061] The memory medium reading device 18 reads data stored in various memory media such as CD (Compact Disc)-ROM, DVD (Digital Versatile Disc)-ROM, Blu-ray Disc, and USB (Universal Serial Bus) memory, and writes data to the memory medium.

[0062] The imaging unit 19 captures and images the object 40 using an imaging function such as a camera mounted on the device, and acquires video and images.

[0063] FIG. 5 is a block diagram showing the hardware configuration of the expert side device 20 according to the present embodiment. As shown in FIG. 5, the hardware configuration of the expert side device 20 overlaps with the hardware configuration of the engineer side device 10 described above, and thus the description thereof is omitted.

[0064] Next, the functional configuration of the information processing apparatus 1 according to the present embodiment will be described. FIG. 6 is a block diagram showing the functional configuration of the information processing apparatus 1.

[0065] As described above, the information processing apparatus 1 according to the present embodiment is mounted on the engineer side device 10 and the expert side device 20. By being mounted on the engineer side device 10 and the expert side device 20, the information processing apparatus 1 according to the present embodiment functions mutually between the respective devices, enabling the transmission, reception, and presentation of various information.

[0066] As shown in FIG. 6, the information processing apparatus 1 includes, as functional configurations, a first acquisition unit 50 and a first presentation unit 51 in the engineer side device 10, and a second acquisition unit 60 and a second presentation unit 61 in the expert side device 20, respectively, in the engineer side device 10 and the expert side device 20. Each functional configuration is realized by the CPU 11 of the engineer side device 10 or the CPU 21 of the expert side device 20 cooperating with each hardware configuration.

[0067] The first acquisition unit 50 acquires an object video showing the object 40 of the instruction.

[0068] The instruction refers to the content when an expert gives an instruction regarding a specific object 40 to an engineer who is at a remote location. For example, in a call center for inquiries about a certain product, such as a home appliance product like an air conditioner or a refrigerator, and a multi-functional device having a copy function, when a user (engineer) wants to inquire about a certain product (object 40) regarding repair, failure, etc., it refers to the content when inquiring to the call center (expert). Also, in this case, it refers to the content when the call center (expert) gives an instruction to a user (engineer) at a remote location regarding the response to repair, failure, etc. of a certain product (object 40).

[0069] FIG. 7 is a diagram showing an example of an object video. In FIG. 7, a smartphone is shown as an example of the engineer-side device 10, and the object 40 is being imaged using the imaging function of a camera (imaging unit 19) mounted on the engineer-side device 10 which is a smartphone. The object video may be a still image showing the object 40, or may be a moving image or a video.

[0070] The object 40 refers to the article that is the object of the instruction. In FIGS. 2 and 3, an object in the form of five connected cylindrical shapes is represented as the object 40, but the object 40 is not limited to an actually existing entity, and may be an object displayed two-dimensionally on a display screen or the like. As long as it can be visually recognized as a video or an image by imaging.

[0071] The second acquisition unit 60 acquires the object video acquired by the engineer-side device 10, and the second presentation unit 61 causes the acquired object video to be displayed on an output device such as a display of the expert-side device 20. Thereby, the expert can confirm the specific object 40 that the engineer at a remote location needs an instruction for.

[0072] The second acquisition unit 60 acquires fingertip information representing the movement of the expert's hand.

[0073] The movement of the expert's hand is the movement of moving the fingertip object 41 displayed by the expert to more clearly indicate the intention of the instruction on the object video, and the fingertip information is the information representing the movement of the expert's hand with respect to the object 40 sensed by the sensor 30.

[0074] FIG. 8 is a diagram showing an example of acquiring the movement of the expert's hand as fingertip information. For example, when it is desired to sense the tapping operation of the index finger of the actual hand 42 of the expert, as shown in FIG. 8, the sensor 30 focuses on the position of the tip of the index finger of the actual hand 42 of the expert. By detecting whether the tip position of the index finger has moved up and down by a certain threshold or more compared to the other fingers of the actual hand 42 of the expert within a certain period of time, the tapping operation of the index finger is sensed.

[0075] The first acquisition unit 50 acquires the fingertip information, and the first presentation unit 51 superimposes and displays the fingertip object 41 represented by the fingertip information on the object video. The display example shown in FIG. 2 is an example of superimposing and displaying the fingertip object 41 on the object video in the engineer-side device 10.

[0076] The second acquisition unit 60 acquires the input information included in the fingertip information or separately input from the expert, and the first acquisition unit 50 acquires the input information included in the fingertip information or separately input from the expert. The first presentation unit 51 changes and presents the expression by the fingertip object 41 according to the input information included in the acquired fingertip information or separately input from the expert.

[0077] The input information separately input by the expert includes not only the fingertip information obtained using the above-described sensor 30, but also input information such as keyboard input, button input, mouse input, or voice. For example, it is information input by keyboard input of a predetermined specific character, predetermined specific button input, contact operations such as taps or touches on a touch panel, or voice of a predetermined word such as "object" or "direction of the screen". The expert includes it in the fingertip information or separately inputs input information as information for clearly indicating whether the object indicated by the fingertip object 41 is the object 40, the direction of the display screen, or other than the object 40.

[0078] The first acquisition unit 50 acquires the input information included in the fingertip information or separately input by the expert, and the first presentation unit 51 causes the expression by the fingertip object 41 to be distinguished according to the input information included in the fingertip information or separately input by the expert and presented to the engineer-side device 10.

[0079] The first presentation unit 51 causes the engineer-side device 10 to present the mode of the fingertip object 41 to be distinguished according to the object indicated by the fingertip object 41 in order to clarify the object indicated by the fingertip object 41 according to the input information included in the fingertip information or separately input by the expert. Examples of the mode of the fingertip object 41 presented by distinguishing according to whether the object indicated by the fingertip object 41 is the object 40, the direction of the display screen, or other than the object 40 include changing the color of the fingertip object 41 itself, presenting an indication line from the fingertip object 41 toward the object being indicated, and presenting a sound when the fingertip object 41 indicates a specific object. Details will be described later.

[0080] In addition, when the object pointed to by the fingertip object 41 is the object 40 according to the input information, the first presentation unit 51 causes the fingertip object 41 to be presented in a predetermined manner. When the object pointed to by the fingertip object 41 is the object 40 based on the input information included in the fingertip information or separately input from an expert, the first presentation unit 51 causes the fingertip object 41 to be presented in a predetermined manner. Examples of presenting the fingertip object 41 in a predetermined manner include changing the color of the fingertip object 41 itself, as described above.

[0081] In addition, the first acquisition unit 50 acquires the movement of the expert's hand as input information, and the first presentation unit 51 presents the expression indicated by the fingertip object 41 by distinguishing according to the mode of the movement of the expert's hand as the acquired input information.

[0082] The mode of the movement of the expert's hand as input information means, for example, when the expert wants to present the object pointed to by the fingertip object 41 as the object 40, sensing a specific hand movement predetermined by the sensor 30. Examples of the mode of the predetermined specific hand movement include a tap operation, finger rotation, etc., which will be described later.

[0083] In addition, the first acquisition unit 50 acquires, as input information, the mode of moving the expert's fingertip up and down as a predetermined mode of the expert's hand movement.

[0084] The mode of moving the expert's fingertip up and down is a so-called tap operation. For example, when the sensor 30 senses that the expert's fingertip moves up and down twice, it is assumed that the object pointed to by the fingertip object 41 is the object 40 in advance. In that case, the first acquisition unit 50 acquires the mode of the expert's fingertip moving up and down twice as input information. The number of times of moving the finger up and down may be appropriately set by the expert or the engineer.

[0085] Further, not limited to the number of times when indicating the object 40, the number of times when indicating the direction of the display screen, etc. may also be set as appropriate. For example, when the sensor 30 senses a mode in which the expert's fingertip moves up and down twice, the object indicated by the fingertip object 41 is set as the object 40, and when the sensor 30 senses a mode in which the expert's fingertip moves up and down three times, the object indicated by the fingertip object 41 is set as the direction of the display screen, and so on.

[0086] Further, the first acquisition unit 50 acquires, as input information, a mode of rotating the expert's fingertip as a predetermined mode of movement of the expert's hand.

[0087] The mode of rotating the expert's fingertip is an operation of drawing a circle with the fingertip. The mode of rotating the fingertip includes cases where the fingertip is moved clockwise and cases where the fingertip is moved counterclockwise. For example, when the sensor 30 senses a mode in which the expert's fingertip is moved clockwise, it is assumed that the object indicated by the fingertip object 41 is the object 40 in advance. In that case, the first acquisition unit 50 acquires, as input information, a mode in which the expert's fingertip moves clockwise. The direction in which the fingertip is rotated can be set as appropriate by the expert or the engineer.

[0088] Further, not limited to the rotation direction when indicating the object 40, the rotation direction when indicating the direction of the display screen, etc. may also be set as appropriate. For example, when the sensor 30 senses a mode in which the expert's fingertip is moved clockwise, the object indicated by the fingertip object 41 is set as the object 40, and when the sensor 30 senses a mode in which the expert's fingertip is moved counterclockwise, the object indicated by the fingertip object 41 is set as the direction of the display screen, and so on.

[0089] Further, the first acquisition unit 50 acquires, as input information, a mode of movement of the expert's hand using a predetermined finger.

[0090] The manner of the movement of the expert's hand using a pre - determined finger means that, for example, when the sensor 30 senses the movement manner using the expert's index finger, it is pre - determined that the object indicated by the fingertip object 41 is the object 40. In that case, the first acquisition unit 50 acquires, as input information, the manner in which the fingertip of the expert's index finger moves. It may be appropriately set by an expert or an engineer which fingertip to use.

[0091] Also, not limited to the finger setting when indicating the object 40, the finger setting when indicating the direction of the display screen, etc. may also be appropriately set. For example, when the sensor 30 senses the movement manner using the index finger, the object indicated by the fingertip object 41 is the object 40, and when the sensor 30 senses the movement manner using the little finger, the object indicated by the fingertip object 41 is the direction of the display screen, etc.

[0092] Also, when the object indicated by the fingertip object 41 is the object 40 according to the number of times the expert's fingertip moves up and down, the first presentation unit 51 causes the fingertip object 41 to be presented in a pre - determined manner.

[0093] Regarding the pre - determined manner of the fingertip object 41 presented by the first presentation unit 51, as will be described later, for example, changing the color of the fingertip object 41 itself, presenting an indication line from the fingertip object 41 toward the object being indicated, presenting by sound when the fingertip object 41 indicates a specific object, etc. can be mentioned.

[0094] Also, the first presentation unit 51 presents, while distinguishing, the expression manner of the tip of the fingertip of the fingertip object 41 displayed overlaid on the object video according to the input information acquired by the first acquisition unit 50.

[0095] The manner of expressing the tip of the fingertip of the fingertip object 41, for example, includes presenting small ripples on the fingertip portion of the fingertip object 41 when the object indicated by the fingertip object 41 is a predetermined specific object.

[0096] FIG. 9 is a diagram showing an example of displaying ripples on the fingertip portion of the fingertip object 41. FIG. 9 represents a display screen on which the fingertip object 41 is displayed superimposed on the object image. Ripples are presented at the tip of the fingertip portion of the fingertip object 41 indicating the object 40 on the display screen in FIG. 9. For example, by presenting ripples at the tip of the fingertip of the fingertip object 41 when indicating the object 40, it is clearly shown that the object indicated by the fingertip object 41 is the object 40. The ripples are not limited to only the case where the fingertip object 41 indicates the object 40. For example, they may be presented when indicating the direction of the display screen, and it may be appropriately set by an expert as to in what cases the ripples are to be presented.

[0097] Also, when the object indicated by the fingertip object 41 is the object 40, the first presentation unit 51 presents an indication line extending from the tip of the fingertip of the fingertip object 41 displayed superimposed on the object image toward the object 40.

[0098] FIG. 10 is a diagram showing an example of presenting an indication line 43 extending from the tip of the fingertip of the fingertip object 41 toward the object 40. FIG. 10 represents a display screen on which the fingertip object 41 is displayed superimposed on the object image. On the display screen in FIG. 10, the fingertip object 41 presenting the indication line 43 extending from the tip of the fingertip toward the object 40 is displayed. By directing the indication line 43 from the tip of the fingertip of the fingertip object 41 toward the object 40, it is clearly shown that the object indicated by the fingertip object 41 is the object 40. The indication line 43 is not limited to only the case where the fingertip object 41 indicates the object 40. For example, it may be presented when indicating the direction of the display screen, and it may be appropriately set by an expert as to in what cases the indication line 43 is to be presented.

[0099] In addition, the first presentation unit 51 changes the presentation mode of the indication line 43 according to the input information. Changes in the line shape, thickness, color, etc. of the indication line 43, and changes in the tip shape of the indication line 43 may be appropriately set by an expert or an engineer. The change in the tip shape of the indication line 43 means, for example, changing the tip shape of the indication line 43 to an arrow shape, a circular shape, or the like. Further, the first presentation unit 51 may change and distinguish the mode of the indication line 43 not only when the object indicated by the fingertip object 41 is the object 40, but also according to the object indicated by the fingertip object 41.

[0100] Furthermore, the length of the indication line 43 may be appropriately changed by an expert. For example, the length of the indication line 43 may be changed by an expert performing an operation such as scrolling the mouse while presenting the indication line 43 from the tip of the fingertip of the fingertip object 41 toward the object 40. Also, when a depth camera, a depth sensor, etc. are mounted on the engineer-side device 10, the first presentation unit 51 may be able to automatically adjust the length of the indication line 43 after detecting the depth, etc. of the distance to the object 40.

[0101] In addition, when the object indicated by the fingertip object 41 is the object 40, the first presentation unit 51 presents it in a distinguishable manner according to the color of the fingertip object 41. When the object indicated by the fingertip object 41 according to the input information acquired by the first acquisition unit 50 is the object 40, it is presented in a distinguishable manner by changing the color of the fingertip object 41 from the color during normal display. For example, when the object indicated by the fingertip object 41 is the object 40, the color of the fingertip object 41 is made red.

[0102] The presentation of the color of the fingertip object 41 is not limited to only when the fingertip object 41 indicates the object 40. For example, even when indicating the direction of the display screen, the color of the fingertip object 41 may be changed or distinguished and presented. Regarding in what cases to change or distinguish and present the color of the fingertip object 41, it may be appropriately set by an expert. As an example, when the object indicated by the fingertip object 41 is the object 40, the color of the fingertip object 41 is made red, and when the object indicated by the fingertip object 41 is the direction of the display screen, the color of the fingertip object 41 is made blue, etc.

[0103] Also, when the object indicated by the fingertip object 41 is the object 40, the first presentation unit 51 presents it by distinguishing with sound. When, according to the input information acquired by the first acquisition unit 50, the object indicated by the fingertip object 41 is the object 40, when displaying the fingertip object 41, it is presented by outputting sounds such as "Pon", "Pipi", "Pii", etc., to distinguish it from the normal presentation mode of the fingertip object 41.

[0104] Another example of the sound may be an audio output such as "Currently, indicating the object". The output of the sound of the fingertip object 41 is not limited to only when the fingertip object 41 indicates the object 40. For example, it may also be output when indicating the direction of the display screen. Regarding in what cases to output the sound together with the display of the fingertip object 41, it may be appropriately set by an expert.

[0105] The second acquisition unit 60 acquires an instruction by an expert or detects the elapse of a predetermined time, and transmits it to the engineer-side device 10. The first acquisition unit 50 acquires the instruction by the expert acquired by the second acquisition unit 60 or the detected elapse of the predetermined time, and the first presentation unit 51 presents it by restoring the change in the presentation mode of the fingertip object 41.

[0106] An instruction by an expert is an instruction to restore and present the mode of presenting the fingertip object 41 to its previous state. The instruction by an expert may be information input in the same way as input information such as predetermined hand movements, keyboard inputs, button inputs, sounds, and voices. Here, the predetermined hand movements include the expert moving his / her own hand to a specific area, and the specific area may be, for example, outside the detection area of the expert-side device 20. That is, in this case, after the expert's hand is no longer detected by the expert-side device 20, when the expert's hand is detected again, it is displayed in the normal state of the fingertip object 41 before the change. Also, when a touch on the touch panel is detected and the presentation mode of the fingertip object 41 is changed, when the touch is no longer detected, it returns to the normal display of the fingertip object 41. When restoring the mode change according to a predetermined time, the predetermined time can be appropriately set by the expert, for example, set to 1 minute after presenting the fingertip object 41 in a changed or distinguishable mode.

[0107] Restoring the presentation of the fingertip object 41 to its previous mode change means returning to the normal display of the fingertip object 41 before presenting it in a changed or distinguishable mode. The first presentation unit 51 returns to the normal display of the fingertip object 41 before the change, without the instruction line 43, etc., for the above-described ripples, presentation of the instruction line 43, and color change of the fingertip object 41.

[0108] Next, taking the engineer-side device 10 including the information processing device 1 according to the present embodiment as the "first device" and the expert-side device 20 as the "second device", the interaction between the first device and the second device will be described. FIG. 11 is a sequence diagram showing the processing of the information processing device 1 according to the present embodiment. The processing of the information processing device 1 according to the present embodiment is executed by the CPU 11 loading and executing an information processing program stored in the storage unit 13, etc., and the CPU 21 loading and executing an information processing program stored in the storage unit 23, etc.

[0109] In step S100, the CPU 11, as the first acquisition unit 50, acquires the object video and transmits it to the second device.

[0110] In step S101, the CPU 21, as the second presentation unit 61, displays the object video on the screen.

[0111] In step S102, the CPU 21, as the second acquisition unit 60, acquires the fingertip information and transmits it to the first device.

[0112] In step S103, the CPU 11, as the first presentation unit 51, displays the fingertip object 41 superimposed on the object video.

[0113] In step S104, the CPU 21, as the second acquisition unit 60, acquires the fingertip information or the input information and transmits it to the first device.

[0114] In step S105, the CPU 11, as the first presentation unit 51, changes the display mode of the fingertip object 41 displayed superimposed on the object video.

[0115] In step S106, the CPU 21, as the second acquisition unit 60, acquires the instruction by the user or detects the elapse of a predetermined time, and transmits it to the first device.

[0116] In step S107, the CPU 11, as the first presentation unit 51, restores the display mode of the fingertip object 41 that was displayed superimposed on the object video and presents it.

[0117] As described above, according to the information processing apparatus 1 according to the present embodiment, when the fingertip object 41 representing the user's instruction is superimposed and displayed on the video of the object 40, the expression by the fingertip object 41 can be presented separately according to the information input by the user. The user in the information processing apparatus 1 according to the present embodiment includes experts and engineers.

[0118] In the present embodiment, the form in which the sensor 30 is connected only to the expert-side device 20 has been described. However, the sensor 30 may be connected not only to the expert-side device 20 but also to the engineer-side device 10. By connecting the sensor 30 to the engineer-side device 10 as well, the movement of the engineer's hand can also be displayed as the fingertip object 41, and communication such as questions from the engineer can be made smoother.

[0119] In the present embodiment, the presentation of the indication line 43 from the tip of the fingertip of the fingertip object 41 toward the object 40 according to the input information has been described. However, the indication line 43 from the tip of the fingertip of the fingertip object 41 toward the object 40 may be always presented, and the mode of the indication line 43 may be changed according to the input information when the input information is acquired.

[0120] In the present embodiment, the distinction and presentation of the object indicated by the fingertip object 41 by sound or voice have been described. However, not limited to the output of sound or voice, the vibration function mounted on the device may be used to present the objects indicated by the fingertip object 41 separately for each object.

[0121] In the present embodiment, the input information separately input from the expert includes not only the fingertip information acquired using the sensor 30 but also the input information by keyboard input, button input, or voice, etc. However, not limited to the actions exemplified above, as another example, moving the fingertip object 41 outside the display screen, opening all the fingers of the expert's actual hand, etc. may be sensed by the sensor 30 and input.

[0122] In this embodiment, the form in which the information processing program is installed in the storage unit 13 and the storage unit 23 has been described, but it is not limited to this. The information processing program according to this embodiment may be provided in a form recorded on a computer-readable storage medium. For example, the information processing program according to this embodiment may be provided in a form recorded on an optical disk such as a CD (Compact Disc)-ROM and a DVD (Digital Versatile Disc)-ROM, or in a form recorded on a semiconductor memory such as a USB (Universal Serial Bus) memory and a memory card. Further, the information processing program according to this embodiment may be acquired from an external device via a communication line connected to the communication I / F 14 and the communication I / F 24.

[0123] In the above embodiment, the processor refers to a processor in a broad sense and includes a general-purpose processor (for example, a CPU: Central Processing Unit, etc.) and a dedicated processor (for example, a GPU: Graphics Processing Unit, an ASIC: Application Specific Integrated Circuit, an FPGA: Field Programmable Gate Array, a programmable logic device, etc.).

[0124] Also, the operation of the processor in the above embodiment may be achieved not only by one processor but also by a plurality of physically separated processors cooperating with each other. Further, the order of each operation of the processor is not limited to the order described in each of the above embodiments and may be changed as appropriate.

Explanation of Signs

[0125] 1 Information processing apparatus 10 Engineer-side device 20 Expert-side device 11, 21 CPU 12, 22 Memory 13, 23 Storage unit 14, 24 Communication I / F 15, 25 Input / Output I / F 16, 26 Input Section 17, 27 Output Section 18, 28 Memory Medium Reader 19 Imaging Section 30 Sensor 40 Object 41 Finger Tip Object 42 Actual Hand 43 Indicator Line 50, 60 Acquisition Section 51, 61 Presentation Section

Claims

1. A processor is provided, wherein the processor acquires an object image reflecting an object to be indicated and fingertip information representing the movement of the user's hand, displays a fingertip object represented by the fingertip information overlaid on the object image, changes and presents the expression by the fingertip object according to whether the object indicated by the fingertip object is the object or the direction of the display screen, as information included in the fingertip information or according to input information input from the user, when the input information is information indicating that the fingertip object indicates the object, changes and presents the expression by the fingertip object, An information processing apparatus.

2. The processor presents the fingertip object in a predetermined manner when the input information is information indicating that the object indicated by the fingertip object is the object. The information processing apparatus according to Claim 1.

3. The input information is the movement of the user's hand, and the processor changes and presents the expression indicated by the fingertip object when the mode of movement of the user's hand is a mode indicating that the fingertip object indicates the object. The information processing apparatus according to Claim 1 or Claim 2.

4. The mode of movement of the user's hand is a mode of moving the user's fingertip up and down, and the processor presents the fingertip object in a predetermined manner when the number of times the user's fingertip moves up and down is the number of times indicating that the fingertip object indicates the object. The information processing apparatus according to Claim 3.

5. The mode of movement of the user's hand is a mode of rotating the user's fingertip, and the processor presents the fingertip object in a predetermined manner when the direction in which the user's fingertip rotates is the direction indicating that the fingertip object indicates the object. The information processing apparatus according to Claim 3.

6. The mode of movement of the user's hand is an operation of contacting an output unit on which the object image is displayed, and the processor presents the fingertip object in a predetermined manner when contacting the output unit. The information processing apparatus according to Claim 3.

7. The input information is the movement of the user's hand using a predetermined finger, The processor is, when the mode of movement of the user's hand using the predetermined finger is a mode for indicating that the fingertip object indicates the object, causing the fingertip object to be presented in a predetermined mode The information processing apparatus according to claim 3.

8. The processor is, in response to the input information, causing the mode of expression of the tip of the fingertip object displayed superimposed on the object image to change and be presented The information processing apparatus according to any one of claims 3 to 7.

9. The processor is, when the input information is information for indicating that the fingertip object indicates the object, causing an indication line to be presented from the tip of the fingertip of the fingertip object displayed superimposed on the object image toward the object The information processing apparatus according to claim 8.

10. The processor is, in response to the input information, changing the mode of presenting the indication line The information processing apparatus according to claim 9.

11. The processor is, when the input information is information for indicating that the fingertip object indicates the object, causing the expression by the fingertip object to change according to the color of the fingertip object and be presented The information processing apparatus according to any one of claims 3 to 7.

12. The processor is, when the input information is information for indicating that the fingertip object indicates the object, causing the expression by the fingertip object to change by sound and be presented The information processing apparatus according to any one of claims 3 to 7.

13. The processor is, based on an instruction by the user or the elapse of a predetermined time, reverting the change in the presentation mode of the fingertip object and presenting it The information processing apparatus according to any one of claims 1 to 12.

14. The instruction by the user is an operation of moving the user's hand to a specific area, The processor is, when detecting the movement of the user's hand to the specific area, reverting the change in the mode of the fingertip object and presenting it The information processing apparatus according to claim 13.

15. The processor is, when the contact with the output unit is released, reverting the change in the mode of the fingertip object and presenting it The information processing apparatus according to claim 6.

16. To a computer, Obtain an object image reflecting the object of the instruction and fingertip information representing the movement of the user's hand. Superimpose and display a fingertip object represented by the fingertip information on the object image. As information indicating whether the object pointed to by the fingertip object is the object or the direction of the display screen, change and present the expression by the fingertip object according to whether it is included in the fingertip information or input information input from the user. When the input information is information indicating that the fingertip object points to the object, change and present the expression by the fingertip object. An information processing program.

17. An information processing system including a first device and a second device, The first device includes a first processor, and the second device includes a second processor. The first processor obtains an object image reflecting the object of the instruction and causes the display unit to display the object image. The second processor obtains fingertip information representing the movement of the user's hand, and transmits the fingertip information to the first device so as to superimpose and display a fingertip object represented by the fingertip information on the object image on the display unit. The first processor changes and presents the expression by the fingertip object on the display unit according to whether the information indicating whether the object pointed to by the fingertip object is the object or the direction of the display screen is included in the fingertip information or input information input from the user. When the input information is information indicating that the fingertip object points to the object, the first processor changes and presents the expression by the fingertip object. An information processing system.

Images